Legacy/Re-Indexing the FSM Data
Overview
In this document, we will learn how to legacy index/re-index the fsm index.
Pre-Requisites
Kubectl access to the required environment in which you want to run the re-indexing
playground pod access
Postman Collection
Steps to legacy-index FSM
After importing the postman collection downloaded from above section, you can find two request
fsm-legacy : This request helps to get the data from fsm/plainsearch api and push data to fsm-enriched topic by indexer service
fsm-legacy-kafkaconnector : This is the request to create a connector which can listen to the fsm-enriched topic and push data to the elastic search with the new index fsm-enriched
Run the fsm-legacy-kafkaconnector request in the playground pod, which would create a connector which would intern start listening to the topic fsm-enriched-sink
Run the fsm-legacy request in the playground pod, which would call indexer service to intiate the process of fetching the data from plainsearch and push the data prepared according to the legacy-index mapping and push the data to the fsm-enriched-sink topic
Whole process would take some time, mean while you can searc for the data in fsm-enriched index in the elastic search
we can go through the logs of the indexer pod, which would help to understand the job is done
Once the job is done, delete the kafka connector running the below curl in the playground
curl --location --request DELETE 'http://kafka-connect.kafka-cluster:8083/connectors/fsm-enriched-es-sink'
Once reindexing is completed, please verfiy the count in fsm index and fsm-enriched index, then delete the fsm index and create alias for fsm-enriched index as fsm.Please use below command for alias creating.
Steps to legacy index VehicleTrip
After importing the postman collection downloaded from above section, you can find two request
vehicleTrip-legacy : This request helps to get the data from vehicletrip/plainsearch api and push data to vehicletrip-enriched topic by indexer service
vehicle-trip-legacy-kafkaconnector : This is the request to create a connector which can listen to the vehicletrip-enriched topic and push data to the elastic search with the new index vehicletrip-enriched
Run the vehicletrip-legacy-kafkaconnector request in the playground pod, which would create a connector which would intern start listening to the topic vehicletrip-enriched-sink
Run the vehicletrip-legacy request in the playground pod, which would call indexer service to intiate the process of fetching the data from plainsearch and push the data prepared according to the legacy-index mapping and push the data to the vehicletrip-enriched-sink topic
Whole process would take some time, mean while you can searc for the data in vehicletrip-enriched index in the elastic search
we can go through the logs of the indexer pod, which would help to understand the job is done
Once the job is done, delete the kafka connector running the below curl in the playground
curl --location --request DELETE 'http://kafka-connect.kafka-cluster:8083/connectors/vehicletrip-enriched-es-sink'
Once reindexing is completed, please verfiy the count in vehicletrip index and vehicletrip-enriched index, then delete the vehicletrip index and create alias for vehicletrip-enriched index as vehicletrip.Please use below command for alias creating.
Steps to legacy index FSM Inbox
Pre-Requisites
Kubectl access to the required environment in which you want to run the re-indexing
playground pod access
Postman Collection
Steps to legacy-index FSM Inbox
After importing the postman collection downloaded from above section, you can find two request
fsm-legacy inbox : This request helps to get the data from fsm/plainsearch api and push data to fsm-application-enriched topic by indexer service
inbox-kafka-connector : This is the request to create a connector which can listen to the fsm-application-enriched topic and push data to the elastic search with the new index fsm-application-enriched
Run the inbox-kafka-connector request in the playground pod, which would create a connector which would intern start listening to the topic fsm-application-enriched-sink
Run the fsm-legacy-inbox request in the playground pod, which would call indexer service to intiate the process of fetching the data from plainsearch and push the data prepared according to the legacy-index mapping and push the data to the fsm-application-enriched-sink topic
Whole process would take some time, mean while you can search for the data in fsm-application-enriched index in the elastic search
we can go through the logs of the indexer pod, which would help to understand the job is done
Once the job is done, delete the kafka connector running the below curl in the playground
curl --location --request DELETE 'http://kafka-connect.kafka-cluster:8083/connectors/fsm-application-enriched-es-sink'
Once reindexing is completed, please verfiy the count in fsm-application index and fsm-application-enriched index, then delete the fsm-application index and create alias for fsm-application-enriched index as fsm-application.Please use below command for alias creating.
Last updated