What's the best way to integrate Airflow with encrypted secrets stored in manifests? #45190
Replies: 1 comment 2 replies
-
|
@potiuk, do you have any recommendations? This would help guiding my future work in the area. Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
No idea. But you should avoid tagging people directly to drag them in discussions. This way you are demanding their exclusive attention to your issue (which is pretty selfish) but also if you are tagging single person, you are limiting your chances of getting help from others because others will think that the tagged person will answer (which is shooting yourself in a foot). Especialy when you expect answer one hour after you posted a question. Remember, that this is a place where people help others for free in their free time, away from familiy and other things they can do, and it's them who decide what and where they can help with and where they put their time and energy. Not mentioning that this is holiday time and you can safely expect that people are currently thinking about something else than Airflow. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry about that :( |
Beta Was this translation helpful? Give feedback.
-
Hello. I hope your day is going well. I'm trying to use ArgoCD to manage Airflow, which makes it tougher to work with secrets, as I can't just check them into the repo due to values being only base64 encoded. I've seen there's AWS Secrets Manager backend, which is a fine option, but has a downside of some manual overhead when updating values. For example,
pgbouncer.iniis a whole config which has some sensitive data (e.g. password in db connection).A solid alternative would be to store the secrets in a source manifests code in encrypted form (e.g. with AWS KMS). This would allow to automate a process bringing up things in a new account/region for example. However, Airflow doesn't seem to support this out of the box. I can set env variables in
airflowLocalSettings(which maps to a Python file run on init), e.g. by creating _ENCRYPTED versions of env variables, decrypting them on startup and setting regular versions of variables. However, it won't work out of the box for things likepgbouncer.ini, which is mounted as a file in the deployment manifest.What's the best way forward? Shall I do some code changes to Airflow to natively support encrypted data with some new backend? Or is there some way to configure all secrets without doing this? Thanks.
Beta Was this translation helpful? Give feedback.
All reactions