This is the second in a series of blogs on dogfooding your own product. The first in this series covered how we approached compliance, and in this one we’ll look at how we went about protecting other critical R&D assets. In later posts, we’ll cover remote access and InfoSec, and finally productivity.
Engineering organizations present numerous challenges for security programs when it comes to remote access. The infrastructure these teams require is varied, ranging from external SaaS to internally hosted web services for wikis, git and build servers, various TCP services such as SSH and RDP, as well as database access and recently a huge wave of Kubernetes. These services are complex and often undocumented, especially as projects are under active development before they reach production environments. Securing these critical R&D assets arguably makes an Engineering Org the most challenging department that InfoSec teams have to manage. In this post, I’ll cover how Banyan Security approached securing remote access for our own R&D organization.
Part II: Protecting Critical R&D assets
Production environments are considered the crown jewels for many attackers and as a consequence most security efforts focus there. Security experts will advise teams to prioritize attack vectors by impact and likelihood and naturally production assets would have the biggest impact if they were breached. That’s where the most defenses are too, so the likelihood of a breach in production may lower because the challenge is greater. Conversely, the impact of a breach in a non-production environment may not have as big an impact and as a result may not have had nearly as much attention paid to them.
As the DevOps era automates everything, we’ve seen a growing threat to attacking the supply and deploy chain in software, be it repositories, database backups used in non-production environments, and other insider attacks on internal tools. Likelihood of a breach in non-prod? Perhaps much higher than production. At Banyan, therefore, it has been a priority to protect all our R&D environments, not just production. And since we’re not using a VPN to establish remote access (for obvious reasons), we’re using our own platform.
Our non-production environments are where R&D happens. It’s the creative playground for prototyping, the infrastructure for core product engineering, the DevOps tools and the QA labs for ensuring the quality of our offering.
This diagram illustrates the differentiated access needed for the different types of roles needed to access our environments.
Non-production means they do not directly contribute to the service we are offering our customers. We have DevOps tools such as Jenkins and GitLab, we have SAST and DAST tools, we have ephemeral on-demand developer environments and long-lived environments. Each of these is composed of a different set of service types and has a different access policy for who should be permitted to consume them.Next, let’s look at user types who need to consume the services:
- Privileged employees (EngineeringInfraTeam below) – small set of trusted employees who have full access to all infrastructure.
- DevOps engineers (EngineeringInfra) – engineers who develop infrastructure and need full access to cloud resources to build and develop the platform.
- Regular engineers (Engineering) – Engineering & QA team members who need to regularly access all lower-tier environments for debugging and troubleshooting and pipeline tools. Jenkins, GitLab, and web, SSH and database access for environments.
- 3rd party devs (PersonalDevices) – contractors who should have access to a limited set of environments, for web and database access.
- Other team members (Everyone) – non-engineers who should have web access to a single environment for pre-release reviews.
These roles embody our least privileged access control model:
We built policies around this access definition, including what level of device trust should be required for each resource. For the employees that have the most powerful access, we require high trust scores. At Banyan, this means EDR running, plus the standard set of device factors all being in place. For these engineers, this would limit using their personal devices for critical resource access unless they are willing to install our EDR protection on it.
We then pursued a series of strategies to implement the access control:
- Access Tier – All long-lived application environments and hosted DevOps tools are placed in a VPC protected by an Access Tier
- SaaS – All SaaS services like GitLab, Datadog, and our public cloud console- are protected by Banyan SaaS
- Secure Remote Access defined in Code – For short-lived environments, we built the service definitions into our DevOps pipeline so that when a new environment is created, it would automatically be available for secure remote access. This was accomplished using our OSS PyBanyan project and some not-yet-released Terraform modules.
Securing R&D is the most challenging of all the departments at Banyan. Engineering has the widest array of technologies, requiring SaaS, hosted web, and various TCP services. These services are continually being modified and re-designed, and the engineers who do so are both finicky and technical. Not supporting them properly runs the risk of introducing “shadow” infrastructures. Fortunately, with Banyan, we have a robust and flexible platform so that the end users are both happy and safe.
End Part II
