CI / CD Pipeline Problems of Large Enterprises
Migrating Thousands of Applications to CI/CD World, Handling different Application Maturity Levels or Handling different Developer Demands can be challenging in Enterprise world.
How can we Handle such seemingly complicated world.
I go with the assumption that we all are aware of Pipeline (Jenkins / Any other); that takes the Code from Code Repository and push it all the way to PRD after going through Several Stages.
This scenario is great (Theoretically or in small Organization). In real enterprise world, there are several complexities that would arise once team decides to move to CI/CD World.
Let’s discuss on some common issues in enterprise world and how those issues can be addressed.
- Hundreds/Thousands of legacy and contemporary applications that would require Migration.
- Applications at different maturity level — enforcing Quality Standards across applications would require sizeable efforts.
- Developers need flexibility to quickly release hot-fix or check the solution quality without pushing it to an Environment.
Applications Migration:
Migrating thousands of applications can be achieved by writing complete pipeline for each Application. That in itself would require several man years though.
Ideal way to achieve this migration is by having a base/library pipeline with the parameters (such as Application Name, Git Hub Location, etc.). Each application can then extend this library pipeline, by just provide values of parameters (few lines max).
This approach should not take more than few lines / application; and would require considerably less time for migration.
Different Maturity Levels
A Big role of CI CD in the enterprise is to enforce the Quality Standards across all applications.
CI-CD Pipeline does that by enforcing Quality checks at various stages of Pipeline. Checks such as Static Code Quality (LINT), Code Coverage, Code SMELLS, Security score etc. are must for any enterprise Applications.
Any deviations from the Standards should should either fail the Build Process or should stop build getting deployed to Production (or Any) environment.
Enforcing Standards is great, but enforcing such Standards equally on all Applications (Legacy vs Contemporary; External Facing vs 1 Internal User Application; Applications using Latest Technologies vs Applications tied up by Archaic Vendor Driven Technology etc.) will not lead to general acceptance.
Ideal approach would be to have Centrally Controlled applications specific configurations. Pipeline should consider application specific configurations while building or deploying application artifacts.
Configuration driven CI-CD approach would enforce tighter restrictions on some, yet be flexible for other applications (different level of restrictions for different parameters).
Ex:
Legacy Application 1 — Static Code Analysis = N/R
Latest Application 1 — Static Code Analysis = No Sev 1, Sev 2 Violations
Legacy Application 1 — Code Coverage = 5%
Latest Application 1 — Code Coverage = 90%
Flexible Pipeline
Sometimes Developer has to release a Hot fix to PRD directly and has no time to go through all the steps on all the environment such as DEV, TST, STG, PPR etc.
Yet sometimes Developer wants to check if its build is good enough for UAT Release. Developer doesn’t want to release to UAT but wants to ensure that the build is good enough from Static Code Quality, Security Vulnerability, Automation Tests etc. perspective.
To handle such scenarios having a strictly defined pipeline would not work. Pipeline should be able to behave differently based on the context or the instructions provided.
One way of making pipeline flexible is Tagging Mechanism.
Developer while checkin in the Code should be able to say #STG as commit comments. Pipeline should deploy the Build from STG onwards.
Developer while Checkin the Code says #TST.Deploy=FALSE; Pipeline should do all the steps (including all the tests) but should revert back before deploying to TST.
One can make its Pipeline as flexible as possible; and that would ensure Pipeline is able to handle anything you through at it (intelligently).
Final Thoughts
There are multiple different set of problems that you would have encountered and solved in your CI-CD Journey.
If you faced any challenges, please share the challenges and how you made your CI-CD world happier world to live in.
