A Path to Green Environment and Sustainable Transportation Ecosystem
As data and analytics become increasingly prominent, business drivers for enterprises and governments for their digital transformation journey, many interesting use cases are evolving from government authorities and city commune/municipality offices asking their suppliers and vendors to provide more data-driven digital solutions.
Business Use Cases and Expected Business Value
One of these use cases is a near-real-time overview of all Electric Vehicles (EV) charging stations within a city so that the city authority can: track the near-real-time progress of the charging station installation program, overall and efficient usage of charging stations; identify peak load-time during the day (thereby enabling the possibility of understanding overall electricity load of the city due to EV charging); as well as plan for future expansion through pattern matching. The interesting part is that the requirements are coming from the city authorities towards their electric charging station installation and maintenance vendors, partners, and service providers as part of the final contract negotiation itself (RFI-Request For Information / RFP-Request For Proposal phase).
Some of the Minimum Viable Products (MVPs) are charging stations planning: city heat-map, identifying and optimising the need for more charge station(s) in a city/location, reducing waiting time for charging, and “Intelligent Charging”. There are huge possibilities of many interesting use cases based on these examples both in B2B (Business-to-Business) and B2C (Business-to-Consumer) segments and enabling a large EV charging ecosystem. Today, advancements in mobile and digital applications provide a basis for finding charger locations in an efficient way.
An ideal use case is where a vehicle driver looking for a charging station is routed to the closest available charging site based on current traffic patterns, distance, connection types, available ports, and current usage and condition of the site. Collecting the data further into a specific EV charging use case would not require additional innovation or new technologies, just leveraging current technologies, data, and analytics.
A Brief Solution Description
We implemented the solution based on our enterprise data platform following Data Mesh architecture (fine-grained, fully governed Data Mesh). The “Data Providing Domains” are Smart Charging Platform (includes charge-points data, asset data, card data, location data etc.), Reference City Map (external), CRM and Billing Domains. The “Data Consuming Domain” is a “sustainable transportation enabler through optimized charging station ecosystem”. Some of the “Data Products” (for consumption and reuse) are charge-point-dataset-static, charge-point-dataset-dynamic, billing-n-payment-dataset-b2b, billing-n-payment-dataset-b2c etc.
High-Level Architectural Design and Governance Model
Fine-grained, fully governed Data Mesh approach was followed to achieve centralized data distribution through the distribution domain and decentralized data consumption and analysis through domain-specific ownership. The Data Mesh implementation model for our use case is like the diagram below.
High-Level Solution Architecture
Although we used the Azure cloud ecosystem to achieve our goal for this specific use case(s), the same can be achieved through other cloud service providers like AWS and GCP without any limitations. A very high-level architecture is presented in the diagram below.
- First, we onboarded the Smart Charging Platform data (“Providing Domain”) to our Enterprise Data Domain by consuming Kafka topics published from the Smart Charging Platform. We used Databricks streaming APIs and Azure Data Factory for trigger based near-real-time data onboarding purposes. Our Enterprise Data Domain is based on the Azure ecosystem, primarily on Azure Data Lake with features like Delta Lake and Lakehouse. Existing legacy systems are also consuming different Kafka topics based on the requirements.
- Next, we processed data in our Data Lake using Databricks notebooks and used Azure Data Factory for scheduling purposes. Data was processed in “Bronze” (Raw), “Silver” (Cleansed) and “Gold” (Curated/Transformed/Consumable) zones. An important consideration of data streaming was until which step we shall continue the streaming process without “breaking” it.
- We decided to push down compute-heavy high data volume processes to the Data Lake (Enterprise Data Domain), and we built “CDM” (Common Data Model) in the “Gold” data zone by implementing data minimization techniques at the source. Here we also had data from other required Data Provider Domains, e.g., CRM, Billing, E-Commerce etc.
- Once the data is available for consumption in the “Gold” zone, we push data to Azure Synapse Analytics in the “Consumer Domain – E-Battery Analytics Layer” for reporting and advanced data analytics and prepare the Data Presentation Layer. We used Microsoft Power BI as Data Presentation Layer and developed a few APIs to be used by third-party web applications for data sharing using Azure API Service. The “E-Battery Analytics Layer” is managed by an agile team that works closely with the business and belongs to the business domain. It’s also possible to enrich data in this layer before presenting it to the presentation layer.
- An Exploratory Data Layer was also prepared as part of the “Enterprise Data Domain”, where data exploration, analysis and future products and services can be developed and tested with de-identified production data.
- We have established a strong data governance process (DataOps) across the whole value chain so that data is available for consumption in a timely and accurate manner and exceptions are logged and managed through the incident management process. Decentralized data management and ownership were one of our primary goals.
- We heavily emphasized data protection and data anonymization processes, including the GDPR aspect, right from the beginning of the use case discussion phase. We adopted the “Security by Design” architecture and data anonymization at the source. Data minimization techniques, data deletion rules and data encryption and masking process were very much discussed during the design phase of the use case.
Sample dashboard for the city heat map where it is possible to see which charging stations are in active state (green), which are not working (red – due to malfunction or some other reasons), which charging stations are planned for future installation (orange) and their installation locations. It also shows whether any one of the ports of a specific charging point is operational or not. The dashboard can be accessed from any device (desktop, laptop, mobile, tablet etc.) by authorized people, and it was also embedded in the customer portal so that external authorities (with guest accounts) can see the near-real-time data update in the heat-map and start planning for future installations based on current usage patterns. The underlying data was also encapsulated into an API and exposed for internal and external usage.
Work closely with business process experts to understand the current state, pain points, and desired target future state. Consider end-to-end implementation and maintenance process, as well as training needs and change management needs in way-of-working. These steps led us to true business value realization.
Use a holistic view to understand the data requirements of the data engineering team, prepare flexible, scalable architecture for automated data onboarding, data processing, data exploration, and data analytics and use automated test scripts wherever possible. Have a team with business SME (Subject Matter Expert), Data engineers, DevOps Engineer, Business Analysts, Architect to work in agile w-o-w.
Reduce the gap between data engineering and data scientists, emphasizing end-to-end automation.
Data became a true asset for organizations around an ecosystem, especially in the areas of emerging technologies. It is fuel for business innovation as the raw material, the finished products, the sales, service, and marketing. The industry around Electric Vehicles charging is growing at a very fast speed, and data is at the centre of the entire value chain supported by modern technologies. There is a huge growth potential in the areas of connected vehicles, smart cities, EV charging stations, green environments and many more. An innovative mindset and business value-focused decentralized data management system with advanced technology and matured agile governance will bring new business opportunities to the table.
About the Author
Prasanta Bhowmik, a customer-focused Enterprise Architect, has expertise in designing and enabling sustainable strategic enterprise architectures, capability and portfolio roadmaps, security and framework for software and modern data management applications in cloud data platform, big data platform and business intelligence area.
He has profound hands-on experience (24+) in driving cloud enablement, conducting enterprise-wide assessments, application design, migration, digital transformation, tools evaluation and solution architecture with advanced analytical systems as well as Enterprise Data Warehouse and data platform in hybrid environments covering various ingestion patterns for large data sets at an Enterprise level.
He has expertise to translate business requirements into IT reference architecture, information models and data visualization to present data insights and actionable items. He is also one of the speakers at the Data Innovation Summit 2023.
The views and opinions expressed by the author do not necessarily state or reflect the views or positions of Hyperight.com or any entities they represent.
Featured image source: https://www.verdict.co.uk/smart-cities-electric-vehicle-revolution/