At the core of all systemic challenges, there are a few problems that hinder a systemic change, limit someone from bringing a change, or cause the system to collapse. The identified pivotal problems are:

Absence of well-defined standards for sanitation

Current standards do not cover all aspects of sanitation and service delivery, for instance, standards of treated human waste, treatment plant technologies, and benchmarks, etc. The ecosystem has created many standards which are not formally notified or enforced.

Where standards exist, awareness and compliance are dismal for four reasons:

• Many actors in the value chain do not have the necessary knowledge, skills, or SOPs

• Complexities in service delivery result in incomplete or improper service.

• A poor requirement specification in RFPs.

• Model Contracts exist but are not followed Delays in corrective action since contracts are not tightly coupled with monitoring

Broken chain of custody from waste generation to reuse

Systemic data either doesn’t exist or remains disjointed to understand how much waste exists, where, when, with whom, and why. FS tends to drop out of the value chain, untreated, it is important to know who dropped it, when, how, or where it ended up polluting the environment. The unavailability of this information hampers corrective and preventive measures.

Availability of verifiable and trusted data at various levels of aggregation to all actors

Data around FS (how much, where, when, who is responsible) is too little, too late. Required data is not created, available, or shared across relevant ecosystem actors Data is not actionable. The performance of sanitation systems remains opaque and unobservable.

India is not poised to convert waste to value, yet

Current systems are not structured to maximize the value from FS and related services. The policy framework doesn’t recognize treated human excreta as compost Unclear and fragmented demand for treated waste contributes to lax operations upstream.

Stakeholder behaviour is often misaligned with safe sanitation practices

• Citizens favour open defecation, construct improper containment units, and ask sanitation workers to clean the tank from inside for extra money.

• Sanitation workers do not actively use PPEs.

• Farmers refrain from using treated waste as manure due to associated stigma.

DISHHA aims to bring benefits for several stakeholders of the ecosystem.

Sanitation Workers and Businesses

• Safe working conditions

• Financially sustainable businesses

• Ability to monitor and optimize services and performance

• Skilling on sanitation practices, standards, and tools for effortless compliance

Citizens

• Easy access through multiple channels to safe and reliable sanitation

• Safe sanitation practices become easy to adopt

• Healthier habitats

• Accountable and transparent services along with grievance redressal

Government

• Improve the coverage of sanitation through scaling FSM

• Ability to create data-driven policies for sanitation

• Rapidly scalable, context-specific, cost-effective digital solutions to drive safe sanitation

• Deskilled tools to provide efficient, quality, and standardized sanitation services

• Ability to monitor, audit, and improve sanitation performance, practices, and standards

• Drive transparency and accountability within the market for sanitation outcomes

• Adopting open-source technologies and avoid vendor lock-in

Market Players

• Innovators will have the ability to build diverse applications in a nascent domain with close feedback from the ecosystem and the platform

• System integrators will have the business opportunity to implement the public digital infra in multiple state-wide implementations

• With standardization, Waste Industry will become efficient to capture value from waste

Civil Society Organizations

• Digitization will create data, process flows, and standards that will become the subject of research and analysis to create new knowledge, technologies and recommendations.

• Knowledge dissemination: the learning from research can contribute to improving policy and standards, creating a feedback loop of do-learn-do to drive the mission impact

Academia

• Digital Infrastructure will provide the civil society the tools to drive Impact

• With data, channels for grievance redressal and ensuring formalization, information to participate in policy-making, CSOs will better prevent harms of social ignorance

Context

Zero deaths, disease, and environmental contamination resulting from poor sanitation

The Sanitation Mission at eGov aims to make this vision a reality for every citizen across India. We believe that the core of the sanitation problem is that current systems fail to deliver against this vision and hence the system itself needs to be progressively reformed to achieve this vision.

Why Now? - A renewed focus on Health and Sanitation

There is high energy in the sanitation ecosystem right now, which has received positive attention in line with the renewed government focus on health in face of the COVID-19 pandemic. In Budget 2021, Health & Wellbeing was clearly the most important focus among the six pillars of the Budget outlined by the finance minister. She announced ₹2.83 lakh crore for the health and wellness sector, an increase of 137% over last year. Similar developments were seen for the sanitation sector such as the announcement of dedicated funding to SBM Urban - 2.0 with a clear focus on complete FSSM in the SBM-Urban and dedicated funding of ₹ 1.41 lakh crore. The development of national and state-level sanitation policies and programs. With over 700+ Faecal Sludge Treatment Plants (FSTPs) being committed, (of which 220 are under construction and 150 are operational), along with commitments from states towards the construction of FSTPs, sanitation has achieved a definitive momentum.

Why us? - Impact at Scale and Speed

The problems of sanitation are systemic in nature, and cannot be solved without the cooperation, coordination, and collaboration of all stakeholders. Such a problem, catalyzing the triple helix model (samaaj, sarkaar, and bazaar), lends itself effectively to digital intervention based on the construct of the societal platform and ecosystem orchestration drawing from our learnings in the urban mission. Given our experience in building public digital infrastructure at scale, we believe we are best placed to create the foundation of this infrastructure and catalyze the ecosystem to progressively solve the most pressing issues and co-create further versions of the digital infrastructure.

Our operating model approach manifests in :

• The platform approach to digital development frees up resources to focus on critical non-technology elements of the sector like creating enabling policy frameworks, change management, capacity building and institutionalization. A shared digital infrastructure approach makes the digital building blocks available to the entire ecosystem, thus, reducing the overall expenditure needed on the technology development side for individual solutions.

• Platforms enable ecosystem actors to develop contextual ‘local’ innovations at speed and scale, thereby accelerating the rate of adoption and proliferation of innovations in the FSM space. This is critical to address challenges in FSM for service delivery and performance management.

• Once the first instance of shared digital infrastructure is created, the digital building blocks become available to the entire ecosystem as open-source. This results in ease of on-boarding for more states and ULBs which aids the rapid expansion of the Infrastructure across geographies.

• Platforms empower ecosystem actors through data by providing access to trusted data in a timely manner at a systemic level. They facilitate data discovery and sharing between disparate urban systems using open standards and APIs. For the sanitation sector, this creates a shared source of truth for all stakeholders to collaborate in problem definition, decision making and shape the growth of the sector based on unified data.

• The opportunity to seed a fundamental shift in the sanitation landscape by demonstrating a radically new approach to digital interventions in sanitation and creating the digital infrastructure that can be useful for FSM work worldview

Given the co-creation, scaling, and interoperability potential of platforms, this approach will enable the sanitation sector to transform itself from a top-down control function to one that enables higher state capacity in areas of core concerns. When seen in consonance with key Government of India moves in creating digital infrastructures in various domains including urban development, payments, health, education, skill development amongst others, as well as the issuance of a strategy on National Open Digital Ecosystems (NODEs), a sanitation platform will be a crucial part of the emerging “governance stack”. Moving forward, the mission will be enabled by critical government policies supporting the adoption of platforms like MeitY’s Strategy on National Open Digital Ecosystems (NODE), India Enterprise Architecture (IndEA), and the National Data Sharing and Accessibility Policy (NDSAP), which are a driving force to create the Shared Digital Infrastructure for the ecosystem.

What is Sanitation?

National Urban Sanitation Policy (NUSP, 2008) defines Sanitation as “safe management of human excreta, including safe confinement treatment, disposal, and associated hygiene-related practices”. While the policy pertains to human excreta, it recognizes that integral solutions need to take account of other elements of environmental sanitation, i.e. solid waste management; generation of industrial and other hazardous wastes; drainage; as also the management of drinking water supply.[1]

Why Sanitation Matters?

“Everyone has the right to life, liberty, and security of person”; Article 21: the fundamental right to life in the Indian constitution embodies the same spirit as the Universal Declaration of Human Rights. If life and the prospect of a healthy and secure life is actually a right of every human being, then lack of safe sanitation is a barrier in achieving this goal. Every year three lakh Indian children die due to Diarrhoea, a preventable disease that happens due to contamination in food and water[1]. The same disease is behind the stunted growth of 4 crore Indian children[2]. A lack of safe sanitation is attributed as a major cause of both, Diarrhoea and Stunting. Poor sanitation denies our fellow citizens the right to clean drinking water. Around 80% of India’s water is severely polluted because people dump raw sewage, silt, and garbage into the country’s rivers and lakes. The water is undrinkable and people rely on illegal and expensive sources of water. [3] While our constitution embraces that “All human beings are born free and equal in dignity and rights”, poor sanitation disproportionately impacts vulnerable communities.

A John Hopkins Study [3] highlights that the deficits in sanitation and the health burden on children is higher in more vulnerable slums. More than a third of households in the lowest income category lack access to any form of safe sanitation[4], and it is not just the case at home. In public institutions, especially in schools, the lack of sanitation facilities burdens the girl child disproportionately once menstrual hygiene becomes a daily concern.[5].

The other key problem is working conditions and safety of sanitation workers According to the numbers collated by the National Commission for Safai Karamcharis (NCSK)[6], a sanitation worker dies every five days, on the job, cleaning sewers and septic tanks across the country. Article 5 of the universal declaration of human rights reads that “No one shall be subjected to torture or to cruel, inhuman or degrading treatment or punishment”, but the constant threat of death mark the life of sanitation workers who work under inhuman and unsafe conditions without personal protection or medical support. Forced by abject poverty and social exclusion, they have no other choice. The ground realities of Sanitation in India force us to examine our respect and concern for human life and dignity. It is no wonder that the man who devoted his entire life to independence, held Sanitation as more important.

Current State of Sanitation Systems in India

The Ministry of Environment, Forest, and Climate Change defines hazardous waste as any waste which due to its physical, chemical, or biological composition is likely to harm health or the environment. By this definition, Human fecal matter is extremely hazardous, the evidence speaks for itself. The high volume and frequency of faecal matter generation, when seen against the current performance of our sanitation system, highlight the urgency of the issue. India ranks 107 out of 143 countries in terms of wastewater treatment according to the estimates of Yale University. [7]

According to gross estimates of the Indian Central Pollution Control Board, our cities generate 72.4 billion litres of sewage every day, of which only 20 billion is treated. It means that every single day, 52.2 billion litres of raw, untreated sewage goes out into the environment.[8] The sewage leeches into the soil, contaminates food and groundwater, causing pollution and deaths.

F most of the toilets in the country are not connected to an underground sewage network. Only 50% of the population used toilets in 2011, and 75% of these toilets were connected to an on-site pit or septic tank where the fecal matter remains contained for 3 to 5 years. These on-site systems are not properly managed, close to 70% of fecal matter ends up contaminating the environment due to poor construction and/or disposal practices. An analysis of Census 2011 shows that overall 93% of faecal matter entered the environment without treatment.

What is FSSM?

The waste we flush down the toilet does not always go into a sewer. Approximately 70% of the households in India have toilets connected to septic tanks or soak pits[8] , technically known as on-site containment systems. They accumulate and store faecal matter over a long period. In sewers, the fecal matter travels daily with a lot of water through long concrete pipes. But in the case of on-site systems, it stays stored for about 3-5 years. Once the storage is full, the waste is emptied and transported to the treatment plant through vacuum trucks. The end-to-end value chain of safe storage, collection, transport, treatment, and end-use or disposal of faecal matter is called Fecal Sludge and Septage management or FSSM. ‘Faecal Sludge’ and ’Septage’ are used to describe faecal matter in a specific physical and chemical state after prolonged storage.

Why does FSSM matter - It is the way forward in Sanitation

FSSM has emerged as a cost-efficient population scale alternative to the networked sewer, which has been the traditional method of wastewater management.

1. Cost-effective: 10 times cheaper than sewer systems. Given the fiscal space for sanitation, FSSM becomes a default method for delivering safe sanitation for all.

2. Coverage: Less than one-third of urban toilet users are connected to sewer systems. The rest are more or less dependent on FSSM systems. Targeting FSSM will help us impact the maximum number of citizens.

3. Scale: In the Swachh Bharat Mission, GoI zeroed down on Open Defecation and constructed 11 crore toilets across the country, most of which are connected to on-site systems, again increasing the need for FSSM.

4. Policy tail-winds: From 2013 to 2021, the Government of India has shown consistent commitment evidenced to Sanitation. Over these years, state governments and union ministries have deeply engaged with domain experts to create several policy enablers in order to accelerate the progress of Sanitation outcomes. With the launch of the Swachh Bharat Mission in 2013, Sanitation came to the forefront of multiple policy interventions including dedicated funding for toilet construction, the proliferation of standards, service-level benchmarks, and Swacchh Survekshan. This was followed by a pivot to a clear focus on FSSM and the importance of interventions across the value chain. The release of the National FSSM Policy in 2017, with further funding allocation across various missions and grants including (AMRUT, SBM, NMCG, 15th Finance Commission), culminating into SBM 2.0 with a clear focus on complete FSSM in the SBM-Urban and funding of 1.41Lakh Crore INR in the budget 2020-21. The States have created momentum with over 700+ Faecal Sludge Treatment Plants (FSTPs) being committed, of which 220 are under construction (with 150 operational plants).

In the next ten years, FSSM will be further mainstreamed in 19 out of 36 states and union territories which cover more than 85% of the Indian population. These states and UTs have followed suit and created state-specific FSSM guidelines and regulatory frameworks to demarcate responsibilities between various bodies at the state and the local level, laying the groundwork for synergizing different schemes to ensure affordable, appropriate, & sustainable FSSM across the state.

By 2030, FSM to serve sanitation needs of 52% of total urban population or ~290mn

1. Market Growth: According to the Toilet Board Coalition (FICCI, Tata Trust, USAID, ADB, etc), the market opportunity of the sanitation economy is at 7.2 Lakh Crore INR in 2021 and will grow to 11 Lakh Crore by 2030.[9] As huge as it is, this number only estimates the market for provision, and operations, and maintenance of toilets which is only the first part of the value chain in FSSM. Currently, citizen-facing services such as construction and emptying of pits and septic tanks are fulfilled by the informal sector. The role of these businesses will be critical to the delivery of safe and sustainable sanitation in the future.

2. Virtuous ecosystem: Compared to the rest of sanitation, FSSM is still a nascent field and malevolent actors such as the waste mafia are not as prevalent or embedded. It is also a smaller ecosystem to orchestrate, and we are already aligned with the ecosystem leaders such as Bill and Melinda Gates Foundation, Centre for Policy Research, WASH Institute, and several key actors in the National FSSM Alliance.

Desludging Operator

Urban Local Body

Treatment Operator

Citizen

Coming soon...

Please feel free to share your feedback/comments on the document here.

A multi-pronged approach, which includes both open platforms and other non-technology components, is critical for creating a larger systemic transformation of the FSM sector. It is important to note that four out of these five steps are non-technology - they include generating demand and adoption, designing programs for impact and sustainability, putting together a coalition of ecosystem actors, building the capacity of the ecosystem to solve at scale, and finally institutionalization of the mission in the federal government.

The five steps of this approach are:

1. Create a shared Narrative and ecosystem coalitions

2. Design and build the foundational national platform

3. Create a scale@speed exemplar of transformation

4. Enable a national-scale open ecosystem to create societal impact

5. Institutionalisation for @ scale-adoption and sustainability

A Shared Narrative and Ecosystem Coalitions

Catalyzing and driving the imagination of national digital infrastructure for the FSM ecosystem will require a focus on building a shared narrative with key stakeholders across industry, academia, civil society, and governments on how a shared digital infrastructure can enable transformation and collaborative problem-solving. This goes beyond advocating new forms of digital technology. eGov uses a co-creative approach to develop its offerings across the spectrum of digital infrastructure (platforms), solutions (applications), and state programs (vehicles for delivering real-world impact).

• While based on our current discussions we have been able to arrive at we need to work with partners like CPR-SCIFI, IIHS, WASH Institute et al to reformulate current approaches to FSM using the digital infrastructure.

• Our approach is to have the primary government-facing partners present evidence-based policy recommendations by themselves until our support is needed. When such support is needed we do join them in the relevant meetings to resolve issues and build consensus.

• The platform approach is amenable to solving a variety of problems across the ecosystem, and hence some of the most wicked sanitation problems will become solvable when addressed through this approach

eGov has a two-pronged approach to communicating these messages:

• Direct to Government Communications - Preferably in coordination with the state’s FSM partners

• Public Forums, Op-Eds, and Conferences - we work closely with a variety of thought leaders to shape conferences on specific topics of governance, create forums for thought leaders to convene, discuss and converge, and ensure that these messages are embedded in public discourse on the topic. We will apply this approach to FSM as well.

Design and build the foundational national platform

This entails deep work with the ecosystem to understand the pivotal problems and how they manifest in various contexts (geography, stakeholders, communities) and abstract the components of the platform that can work across India. The key areas of work are :

• The design of various layers of the platform: Core Data Infrastructure, Core Services, and Applications layer.

  • Core Data Infrastructure layer forms the base of the stack and comprises data registries, a shared source of truth, and data infrastructure

  • Core Services layer that houses context-neutral functional services which can be leveraged to build sanitation solutions

  • Applications layer provides context-sensitive solutions required to implement large-scale sanitation programs

• Development of key building blocks of the platform (principles, standards, and digital artefacts). These are the more fundamental context-invariant components of the shared digital infrastructure.

• Build reference solutions (applications) on the platform for the most pressing problem

Create a scale@speed exemplar of transformation

In addition to conceptual thinking, it is crucial to demonstrate the approach in action. This will involve working with one lead state to build a reference implementation working with the local ecosystem and market players. This will be designed, tracked, and implemented, so that the interplay of FSM policies, institutions, and technology can be captured effectively and used to illustrate the approach in practice. The idea here is to drive re-imagination and accelerate adoption by demonstrating value creation and impact. This will also strengthen the evidence-based recommendations (from concept to actual results) and accelerate the national policy level push. In addition, it creates a pull in the market for commercial players to participate.

Enable a national-scale open ecosystem to create societal impact

Participation of the market players in the national rollout and building innovations is critical for sustainability and scale. This creates the requisite system capacity and provides impetus to the market forces to build local solutions. We will work with commercial enterprises as well as local players and entrepreneurs in helping them build local solutions on the FSM platform. We believe the key principles of engaging with the ecosystem are to ensure open and inclusive systems that make it easier for all stakeholders to participate, coordinate, and communicate; to build trust between stakeholders to improve the quality of both the interactions and the outcomes achieved; and to champion a culture of data-driven decision-making among all stakeholders, supported by data from the platform itself.

Institutionalisation for @ scale-adoption and sustainability

For the long-term sustainability of the initiative, it is important to institutionalize the shared digital infrastructure in a suitable public institution which has the locus standi and the capabilities to make it available at a national scale. Given the glacial process by which governments create the needed institutional architecture and capabilities, we will undertake effort for this early on.

Our current awareness and understanding of the problems in septage management are based on our interactions with ecosystem leaders and leading reports, whitepapers, and published materials. We are deeply aware that FSSM systems have highly interdependent parts. Each stage of the FSSM value chain impacts how effectively the next stage functions.

For instance, if the septic tanks do not have proper access, they will add to the cost of emptying, adding a time cost burden on the desludging operators (DSOs). Similarly, it becomes unviable for the DSOs to dump the waste at the treatment plant if it is far from the city which is often the case. Treatment plants are located as per administrative or financial concerns; locations where public land is available or where surroundings do not have residences. This shows how merely one factor can impact the final sanitation outcome if not considered comprehensively. If the location is not conducive for DSOs to supply the waste to the plant or the time and distance are not factored into the cost, it will remain profitable to dump the waste in the open and go for the next service. The delivery of safe sanitation requires the infrastructure and services to work in tune with each other. Calibration in the different parts of the value chain is critical to achieving functional sanitation systems.

While the linear sanitation value chain gives a lucid frame to understand the ideal flow of FS, it doesn’t capture the variety of stakeholders in the interactions that control and influence the current effectiveness of sanitation systems.

Systems Map

In the following section, we use a systems mapping approach to delineate the various factors and interactions within the sanitation value chain. We will explore various systemic challenges at each part of the value chain which finally culminate into the poor sanitation outcomes.

Part 1 - Toilets and Containment Units

City Managers, Centre/State Govt

While Sanitation has been a clear priority for the Centre and State Governments, city administrators have been strapped due to deficiencies in standards, building codes, municipal processes, and contracting and monitoring capabilities which makes it difficult to ensure the adequacy, usability, and safety of the toilets provisioned by public funds.

Masons

The role of masons is fundamental in proper construction of toilets and containment units. Currently, a majority of masons on construction standards. They are rarely employed using formal contracts which contributes to the lack of traceability and accountability when it comes to the construction of toilets.

Citizens

There is a general concern among citizens about their health and pollution of the environment, but they are unaware of the impacts of improper construction of toilets and containment tanks. Owing to this lack of awareness or constraints of affordability and space, citizens influence the masons to construct containment units that do not follow the technical standards. It is difficult to identify trained masons and the procurement of service happens through offline social networks and word of mouth.

Part 2 - Emptying and Transport

Desludging Operators (DSOs)

• Lack of proper access to the containment systems adds time-cost to service.

• Treatment plants are located far from the city which makes safe dumping unviable.

• While the payment from citizens is a clear incentive for emptying the tanks or pits there are no such incentives that ensure the safe transport of the faecal sludge to the treatment plant.

• Operations remain inefficient without timely and useful information for service delivery (location, house width, tentative volume).

• The working conditions are highly risky, provisions for personal protection equipment and emergency healthcare services are rare and sparse.

Citizens

• It is difficult to identify and book desludging services and service delivery is not reliable.

• Low-income households are denied service since they are not able to afford the full cost.

City Managers, Centre/State Govt

• Difficult to regulate the market for safe dumping without compromising service delivery.

• There is no clear and actionable information in terms of safe or unsafe dumping of FS.

• Lack of necessary pricing policies, service, and infra standards, licensing processes, and contracting and monitoring capacity limit the ability of decision-makers to take action.

Part 3 - Treatment

City Managers, Centre/State Govt

• Difficult to regulate the market for across various stages of construction and operations

• Operational data of the treatment plant and process is often recorded offline and used for post-facto auditing. There is a pervasive lack of actionable information.

• Lack of rational pricing policies, comprehensive service benchmarks and infrastructure standards, contracting and monitoring tools hamper corrective action.

Plant Operators

• The supply of desludging load varies in an unreliable and unpredictable manner. This variance shocks in the treatment capacity, short-circuits the treatment process, and over a long term leads to the failure of the system.

• Treatment plant management and maintenance is difficult and costly, the payments are often not linked to performance. The performance is not directly causally linked to the environmental impact.

• Operations are not monitored to facilitate preventive action within the plant, and lack any binding linkages with standard operating procedures and service level agreements.

Part 4 - Ecosystem

Civil Society and Academia

• Researchers struggle without the availability of systemic data to create new knowledge around the failures, risks, and opportunities to give recommendations for improvement.

• Since policy and standards are not mapped to operational data, it is difficult for the ecosystem to translate knowledge to action to impact and access.

Government

• Inability to trace the impact and proper usage of grants and subsidies for sanitation

• Limited state capacity in terms of budget, skilled human resources, tools, and technology render sanitation a highly complex problem for sanitation departments

• Coordination across multiple functions is needed to keep sanitation systems functional: standard-setting, policy-making, contracting, audit, monitoring among many others

• Limited penetration of technology, innovation, and competition in the sanitation sector, without transparency it remains difficult for the government to enforce accountability across internal processes and market interactions

What is DISHHA?

Digital Infrastructure for Sustainable and Healthy Habitats is drawn from the principles of societal platforms leading us into imagining a space where sanitation has shared resources, curated and evolving knowledge, and unified yet contextual solutions that speak to the needs of the community. We believe every changemaker must have access to a unified, shared, and context-invariant digital infrastructure — a unified software, hardware & network solution that aggregates, secures, transfers, and operates information from multiple physical touchpoints. This enables the ecosystem to use these digital building blocks like standard lego-pieces and build context-dependent solutions. Plug-and-play blocks not only free up resources that can be used for focusing on critical non-technology elements like policy frameworks, capacity building, change management but also avoids a chaotic, siloed, and unsupervised digital space.

Principles

The platform is being built as a Public Digital Good and follows a key set of design principles

• Single Source of Truth

• Federated

• Minimum

• Privacy and Security

• Performance at Scale

• Open

Goals

Ensure traceability of waste by enabling the ecosystem with

• Digital standards for sanitation

• Chain of Custody

• Actionable Data

• Code for Innovation

to move the habitats towards zero untreated waste

Pillars and Impact

The Mission will holistically look at the Pillars of Sanitation and the Impact across the pillars

Addressing Pivotal Problems in FSSM

Our approach to addressing pivotal problems:

Absence of well-defined standards for sanitation

Engaging with the ecosystem, the mission will aim to devise standards and enable compliance for infrastructure and processes in the waste value chain.

• Standards of physical infrastructure: The platform will incorporate existing standards and set new standards in the infrastructure so that any digital solution can easily adhere to the standards that cause an impact on the sanitation sector

• Standard processes and contracts: Within the area of control, the platform will have built-in templatized workflows that can enable ease of compliance by deskilling and training. Besides, the platform also enables recording skill and certification levels of actors

Broken chain of custody from waste generation to reuse

The digital infrastructure will enable to answer questions like “how did it end up here?“, “what is the chain of custody of this waste unit?“, “how much ended up in the environment?“, etc.

• The platform will assign a unique identifier to entities and people along the waste value chain. It will make provisions in the processes to capture waste information that enables traceability along its journey through the value chain.

• In addition to making provisions to capture information throughout the process, the platform will also strive towards the adoption of reference applications and enabling extended solutions to create the data required for the reliable chain of custody of faecal sludge

Availability of verifiable and trusted data at various levels of aggregation to all actors

• The verified data of all the entities and the non-repudiable transaction trails shall lead to higher trust and stronger accountability. The platform will be data-driven and will use data generated through transactions for reporting and analysis.

• Public open data shall be made available via APIs for transparency. The access to open data will ensure high-quality analytics, shorter cycles for system improvement, and most importantly, enabling changemakers to become highly responsive to user and market needs

India is not poised to convert waste to value, yet

• The platform will capture waste transaction data and enable ensuring the quality of treated waste materials. This will help to understand waste demand and supply.

• Besides, the platform will orchestrate matching of demand and supply, setting standards for treated waste, and defining data-driven policies to enable moving towards sustainability.

Stakeholder behaviour is often misaligned with safe sanitation practices

• Along with the ecosystem, the mission will aim to understand and design interventions to drive behavioral change for the usage of toilets by citizens, the entrepreneurism of treatment operators, and generating demand for treated biosolids.

• Using data, the changemakers will be enabled to understand the gaps between provisions and usage and devise suitable interventions to bridge the gaps.

Outcomes

While addressing the pivotal problems, Mission will focus on driving these outcomes:

Pre-read:

• Know the basics of Kubernetes: https://www.youtube.com/watch?v=PH-2FfFD2PU&t=3s

• Know the basics of kubectl commands

• Know kubernetes manifests: https://www.youtube.com/watch?v=ohSUtEfDefc

• Know how to manage env values, secrets of any service deployed in kubernetes https://www.youtube.com/watch?v=OW244LxB4oI

• Know how to port forward to a pod running inside k8s cluster and work locally https://www.youtube.com/watch?v=TT3nd5n5Yus

• Know sops to secure your keys/creds: https://www.youtube.com/watch?v=DWzJ87KbwxA

1. Choose the Cloud

Choose you cloud and follow the Instruction to setup a Kubernetes cluster before moving on to the Deployment.

2. Deployment

3. Destroy the Cluster

Finally, cleanup the cluster Setup if you wish, using the following command. This will delete the entire cluster and other cloud resources that were provisioned for the DIGIT Setup.

cd DIGIT-DevOps/infra-as-code/terraform/my-digit-eks
terraformdestroy​

Conclusion:

All Done, we have successfully Created infra on Cloud, Deployed Digit in the cluster.

Coming soon ⌛

The Amazon Elastic Kubernetes Service (EKS) is one of the AWS services for deploying, managing, and scaling any distributed and containerized workloads, here we can provision the EKS cluster on AWS from ground up and using an automated way (infra-as-code) using terraform and then deploy the DIGIT Services config-as-code using Helm.

Pre-read:

• Know about EKS: https://www.youtube.com/watch?v=SsUnPWp5ilc

• Know what is terraform: https://youtu.be/h970ZBgKINg

Prerequisites

1. ​AWS account with the admin access to provision EKS Service, you can always subscribe to free AWS account to learn the basics and try, but there is a limit to what is offered as free, for this demo you need to have a commercial subscription to the EKS service, if you want to try out for a day or two, it might cost you about Rs 500 - 1000. (Note: Post the Demo, for the internal folks, eGov will provide a 2-3 hrs time bound access to eGov's AWS account based on the request and available number of slots per day)

2. Install kubectl on your local machine that helps you interact with the kubernetes cluster

3. Install Helm that helps you package the services along with the configurations, envs, secrets, etc into a kubernetes manifests

4. Install terraform version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph and also it helps to destroy the cluster at one go.

5. ​Install AWS CLI on your local machine so that you can use aws cli commands to provision and manage the cloud resources on your account.

6. Install AWS IAM Authenticator that helps you authenticate your connection from your local machine so that you should be able to deploy DIGIT services.

7. Use the AWS IAM User credentials provided for the Terraform (Infra-as-code) to connect with your AWS account and provision the cloud resources.

   ​

   1. You'll get a Secret Access Key and Access Key ID. Save them safely.

   2. Open the terminal and Run the following command you have already installed the AWS CLI and you have the credentials saved. (Provide the credentials and you can leave the region and output format as blank)

   Copy

   aws configure --profile egov-workshop-account 
   
   AWS Access Key ID []:<Your access key>
   AWS Secret Access Key []:<Your secret key>
   Default region name []: ap-south-1
   Default output format []: text

   1. The above will create the following file In your machine as /Users/.aws/credentials

   Copy

   [egov-test-account] 
   aws_access_key_id=*********** 
   aws_secret_access_key=****************************

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy DIGIT. The following picture shows the various key components. (EKS, Worker Nodes, Postgress DB, EBS Volumes, Load Balancer)

EKS Architecture for DIGIT Setup

Considering the above deployment architecture, the following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every env, it'll have the same infra.

• EKS Control Plane (Kubernetes Master)

• Work node group (VMs with the estimated number of vCPUs, Memory)

• EBS Volumes (Persistent Volumes)

• RDS (PostGres)

• VPCs (Private network)

• Users to access, deploy and read-only

Understand the Resource Graph in Terraform script:

• Ideally, one would write the terraform script from the scratch using this doc.

• Here we have already written the terraform script that provisions the production-grade DIGIT Infra and can be customized with the specified configuration.

• Let's Clone the DIGIT-DevOps GitHub repo where the terraform script to provision EKS cluster is available and below is the structure of the files.

git clone --branch release https://github.com/egovernments/iFix-DevOps.git
cd iFix-DevOps/infra-as-code/terraform


└── modules
    ├── db
    │   └── aws
    │       ├── main.tf
    │       ├── outputs.tf
    │       └── variables.tf
    ├── kubernetes
    │   └── aws
    │       ├── eks-cluster
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       ├── network
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       └── workers
    │           ├── main.tf
    │           ├── outputs.tf
    │           └── variables.tf
    └── storage
        └── aws
            ├── main.tf
            ├── outputs.tf
            └── variables.tf

Example:

• VPC Resources:

  • VPC

  • Subnets

  • Internet Gateway

  • Route Table

• EKS Cluster Resources:

  • IAM Role to allow EKS service to manage other AWS services

  • EC2 Security Group to allow networking traffic with EKS cluster

  • EKS Cluster

• EKS Worker Nodes Resources:

  • IAM role allowing Kubernetes actions to access other AWS services

  • EC2 Security Group to allow networking traffic

  • Data source to fetch latest EKS worker AMI

  • AutoScaling Launch Configuration to configure worker instances

  • AutoScaling Group to launch worker instances

• Database

  • Configuration in this directory creates set of RDS resources including DB instance, DB subnet group, and DB parameter group.

• Storage Module

  • Configuration in this directory creates EBS volume and attaches it together.

  ​

• The following main.tf with create s3 bucket to store all the state of the execution to keep track.

  iFix-DevOps/Infra-as-code/terraform/sample-aws/remote-state

  main.tf​

provider "aws" {
  region = "ap-south-1"
}

#This is a bucket name that you can name as you wish
resource "aws_s3_bucket" "terraform_state" {
  bucket = "try-workshop-yourname" 

  versioning {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

#This is a bucket name that you can name as you wish
resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "try-workshop-yourname"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

1. The following main.tf contains the detailed resource definitions that need to be provisioned, please have a look at it.

   Dir: iFix-DevOps/Infra-as-code/terraform/sample-aws

   main.tf

# master configs, terraform state helps to maintain the flow of the execution
terraform {
  backend "s3" {
    bucket = "try-workshop-yourname"
    key = "terraform"
    region = "ap-south-1"
  }
}

# Network Module
module "network" {
  source             = "../modules/kubernetes/aws/network"
  vpc_cidr_block     = "${var.vpc_cidr_block}"
  cluster_name       = "${var.cluster_name}"
  availability_zones = "${var.network_availability_zones}"
}

# PostGres DB
module "db" {
  source                        = "../modules/db/aws"
  subnet_ids                    = "${module.network.private_subnets}"
  vpc_security_group_ids        = ["${module.network.rds_db_sg_id}"]
  availability_zone             = "${element(var.availability_zones, 0)}"
  instance_class                = "db.t3.medium"
  engine_version                = "11.5"
  storage_type                  = "gp2"
  storage_gb                    = "100"
  backup_retention_days         = "7"
  administrator_login           = "egovdev"
  administrator_login_password  = "${var.db_password}"
  db_name                       = "${var.cluster_name}-db"
  environment                   = "${var.cluster_name}"
}

# **********  Uses with various access like Admin, Dev, Deploy
module "iam_user_deployer" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-user"

  name          = "${var.cluster_name}-kube-deployer"
  force_destroy = true  
  create_iam_user_login_profile = false
  create_iam_access_key         = true

  # User "egovterraform" has uploaded his public key here - https://keybase.io/egovterraform/pgp_keys.asc
  pgp_key = "${var.iam_keybase_user}"
}

module "iam_user_admin" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-user"

  name          = "${var.cluster_name}-kube-admin"
  force_destroy = true  
  create_iam_user_login_profile = false
  create_iam_access_key         = true

  # User "egovterraform" has uploaded his public key here - https://keybase.io/egovterraform/pgp_keys.asc
  pgp_key = "${var.iam_keybase_user}"
}

module "iam_user_user" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-user"

  name          = "${var.cluster_name}-kube-user"
  force_destroy = true  
  create_iam_user_login_profile = false
  create_iam_access_key         = true

  # User "test" has uploaded his public key here - https://keybase.io/test/pgp_keys.asc
  pgp_key = "${var.iam_keybase_user}"
}



# ********** EKS Cluster (Control Plane) **********
data "aws_eks_cluster" "cluster" {
  name = "${module.eks.cluster_id}"
}

data "aws_eks_cluster_auth" "cluster" {
  name = "${module.eks.cluster_id}"
}
provider "kubernetes" {
  host                   = "${data.aws_eks_cluster.cluster.endpoint}"
  cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)}"
  token                  = "${data.aws_eks_cluster_auth.cluster.token}"
  load_config_file       = false
  version                = "~> 1.11"
}

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = "${var.cluster_name}"
  cluster_version = "${var.kubernetes_version}"
  subnets         = "${concat(module.network.private_subnets, module.network.public_subnets)}"

  tags = "${
    map(
      "kubernetes.io/cluster/${var.cluster_name}", "owned",
      "KubernetesCluster", "${var.cluster_name}"
    )
  }"

  vpc_id = "${module.network.vpc_id}"

  # ********** Worker Nodes
  worker_groups_launch_template = [
    {
      name                    = "spot"
      subnets                 = "${concat(slice(module.network.private_subnets, 0, length(var.availability_zones)), slice(module.network.public_subnets, 0, length(var.availability_zones)))}"
      override_instance_types = "${var.override_instance_types}"
      asg_max_size            = 4
      asg_desired_capacity    = 4
      kubelet_extra_args      = "--node-labels=node.kubernetes.io/lifecycle=spot"
      additional_security_group_ids = ["${module.network.worker_nodes_sg_id}"]
      spot_allocation_strategy= "capacity-optimized"
      spot_instance_pools     = null
    },
  ]

  map_users    = [
    {
      userarn  = "${module.iam_user_deployer.this_iam_user_arn}"
      username = "${module.iam_user_deployer.this_iam_user_name}"
      groups   = ["system:masters"]
    },
    {
      userarn  = "${module.iam_user_admin.this_iam_user_arn}"
      username = "${module.iam_user_admin.this_iam_user_name}"
      groups   = ["global-readonly", "digit-user"]
    },
    {
      userarn  = "${module.iam_user_user.this_iam_user_arn}"
      username = "${module.iam_user_user.this_iam_user_name}"
      groups   = ["global-readonly"]
    },    
  ]
}

# ********** EBS Volumes for the statefulsets (PVCs)
module "es-master" {

  source = "../modules/storage/aws"
  storage_count = 3
  environment = "${var.cluster_name}"
  disk_prefix = "es-master"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "2"

}
module "es-data-v1" {

  source = "../modules/storage/aws"
  storage_count = 3
  environment = "${var.cluster_name}"
  disk_prefix = "es-data-v1"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "25"

}

module "zookeeper" {

  source = "../modules/storage/aws"
  storage_count = 3
  environment = "${var.cluster_name}"
  disk_prefix = "zookeeper"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "2"

}

module "kafka" {

  source = "../modules/storage/aws"
  storage_count = 3
  environment = "${var.cluster_name}"
  disk_prefix = "kafka"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "50"

}

Custom variables/configurations:

You can define your configurations in variables.tf and provide the env specific cloud requirements so that using the same terraform template you can customize the configurations.

├── iFix-dev
│   ├── main.tf 
│   ├── outputs.tf
│   ├── providers.tf
│   ├── remote-state
│   │   └── main.tf
│   └── variables.tf

Following are the values that you need to mention in the following files, the blank ones will be prompted for inputs while execution.

​variables.tf ​

## Add Cluster Name
variable "cluster_name" {
  default = "<Desired Cluster name>"  #eg: my-digit-eks
}

## Add vpc_cidr_block
variable "vpc_cidr_block" {
  default = "CIDR" 
}

# If you want prod grade N/W, you can define HA, DRS with multi zone
variable "network_availability_zones" {  
  default = ["ap-south-1b", "ap-south-1a"]
}

# Which zone, it matters
variable "availability_zones" {
  default = ["ap-south-1b"]
}

variable "kubernetes_version" {
  default = "1.18"
}

# instance type for your worker nodes like r5a.large is 8 vCPU and 16GB RAM
variable "instance_type" {
  default = "r5a.large"
}

# spot instance configuration
variable "override_instance_types" {
  default = ["r5a.large", "r5ad.large", "r5d.large", "t3a.xlarge"] 
}

# number of machines as per estimate
variable "number_of_worker_nodes" {
  default = "3"
}

##Add ssh key in case you want to ssh to nodes
variable "ssh_key_name" {
  default = "ssh key name"
}

# terraform users ssh public key, you need to one for you, refer below to create yours
variable "iam_keybase_user" {
  default = "keybase:egovterraform"
}

# will be prompted to provide during the execution
variable "db_password" {}

​

Important: Create your own keybase key before you run the terraform

• Use this URL https://keybase.io/ to create your own PGP key, this will create both public and private key in your machine, upload the public key into the keybase account that you have just created, and give a name to it and ensure that you mention that in your terraform. This allows to encrypt all the sensitive information.

  • Example user keybase user in eGov case is "egovterraform" needs to be created and has to uploaded his public key here - https://keybase.io/egovterraform/pgp_keys.asc​

  • you can use this portal to Decrypt your secret key. To decrypt PGP Message, Upload the PGP Message, PGP Private Key and Passphrase.

Run terraform

Now that we know what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to your env.

Let's begin to run the terraform scripts to provision infra required to Deploy DIGIT on AWS.

1. First CD into the following directory and run the following command 1-by-1 and watch the output closely.

cd DIGIT-DevOps/infra-as-code/terraform/sample-aws/remote-state
terraform init
terraform plan
terraform apply


cd DIGIT-DevOps/infra-as-code/terraform/sample-aws
terraform init
terraform plan
terraform apply

Upon Successful execution following resources gets created which can be verified by the command "terraform output"

• s3 bucket: to store terraform state.

• Network: VPC, security groups.

• IAM users auth: using keybase to create admin, deployer, the user. Use this URL https://keybase.io/ to create your own PGP key, this will create both public and private key in your machine, upload the public key into the keybase account that you have just created, and give a name to it and ensure that you mention that in your terraform. This allows to encrypt all the sensitive information.

  • Example user keybase user in eGov case is "egovterraform" needs to be created and has to uploaded his public key here - https://keybase.io/egovterraform/pgp_keys.asc​

  • you can use this portal to Decrypt your secret key. To decrypt PGP Message, Upload the PGP Message, PGP Private Key and Passphrase.

• EKS cluster: with master(s) & worker node(s).

• Storage(s): for es-master, es-data-v1, es-master-infra, es-data-infra-v1, zookeeper, kafka, kafka-infra.

• Use this link to get the kubeconfig from EKS to get the kubeconfig file and being able to connect to the cluster from your local machine so that you should be able to deploy DIGIT services to the cluster.

aws sts get-caller-identity

# Run the below command and give the respective region-code and the cluster name
aws eks --region <region-code> update-kubeconfig --name <cluster_name>

1. Finally, Verify that you are able to connect to the cluster by running the following command

kubectl config use-context <your cluster name>

kubectl get nodes

NAME                                             STATUS AGE   VERSION               OS-Image           
ip-192-168-xx-1.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-2.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-3.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-4.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2

Whola! All set and now you can go with Deploy Product..