PenTesticles

The popularity of Bug Bounties has grown hugely over the last 3-5 years, with start-ups taking advantage of eager VCs and Angel investors who want a piece of the ‘Cyber’ pie. As this niche market has become more mature, growth opportunities for crowd sourced frameworks have peaked and pressures from investors to diversify have amplified. As a result, we now see emboldened assertions from some providers that their programs replace services such as penetration testing. For me, this is quite a stretch and I’m going to discuss the reasons why in this post, with some input from some of the world’s top bounty hunters.

I’d like to state from the off, that this isn’t an attack on Bug Bounties as a notion, but a counter to marketing by organisations in this space that misrepresent what these types of programs can logically provide.

As a Senior Director at one of the world’s largest pen testing practices, I’m keen to stay as objective as possible, so I decided to reach out to a few prolific bug hunting friends to get their opinions and to keep me honest. Please note that my views are my own and not that of my employer. A huge thanks to Yappare, Shubham Shah and John ‘n0x00’ Carroll for their inputs. In order to give you some background of the quality of their inputs and why you should listen to these guys, I’ve included a quick overview of the contributors to this post. Yappare is currently (at time of writing) ranked 2nd All-time on BugCrowd and is the author of a fantastic blog, that can be found here. John Carroll is currently an AppSec gun-for-hire and one of the original Top 10 bug bounty hunters for Bugcrowd back in 2013, he was also one of the most prolific charity bounty hunters around. John’s blog is really well known for its humour and innovative techniques, check it out here. Shubham Shah is regarded as one of the most innovative bug hunters in the world, regularly speaking at conferences about his tools and techniques to make the most out of ‘bounty time’, he’s also fairly prolific on HackerOne. Check out his blog here; his post on 120 bugs in 120 days is particularly cool.

Key Points

I think it’s best to start the post with some key points about Bug Bounties that probably aren’t clear or talked about by the companies themselves. However, they’re very relevant when you’re trying to decide whether this type of program is for you. Hopefully, this will provide a behind-the-scenes view of how ‘the crowd’ actually works.

A huge proportion of the top bug bounty hunters are Penetration testers by day, who are employed by Security consultancies. According to Shubs, “a large amount of the top bug hunters on Bug Crowd and H1 are Penetration testers or security engineers for consultancies or in-house”. Looking closely at the leaderboards for other similar programs, it’s clear this is the case across the industry. Therefore, claims of better standards of testing or more technical expertise versus consultancies are generally unfounded. When you utilise a Bug Bounty company, you’re essentially calling upon a lot of the same resources who happen to use their downtime to find your bugs. The consultancies they work for also provide most of – if not all – the training for these guys too, so that’s where the expertise comes from.
A typical bounty approach does not cover a full pen testing methodology. I understand that some Bug Bounty companies have private bounties or additional programs that offer ‘full methodology testing’ and in honesty I haven’t had first-hand experience of the outputs. However, this sounds suspiciously like a standard bug bounty (with arguable more successful testers included by ‘invite only’) and penetration testing respectively. I’m sure it costs a lot more though. As the general premise of a bounty program is to only pay out on findings, this directly affects the behavior and approach of their testers. The upshot of this is that bug hunters tend to target critical and high risk issues (as they pay the most) and if they don’t find them within an arbitrary time window, they move on to the next bounty. Time is money for bug hunters and there’s other sites and other things to do at evenings and weekends. This means that coverage is not a priority when compared to a typical Penetration testing approach. Often, in a Pen test, lower risk issues can be chained to produce a higher risk finding – testers also frequently work in pairs and combine results. When you consider a crowd sourced approach where the players are working in isolation against each other, there isn’t the possibility of combining vulnerabilities as the other hunters cannot see all the issues raised while the program is running – nobody is tracking the big picture.
You’re competing with other firms for tester time. Shubs made a really good point on this topic when I asked him about his methodology, he said “I pick out my bug bounty programs like a company would pick their ideal clients”. This is a really interesting point, as there is a lot of kudos associated within the Bug Bounty community for finding bugs in certain sites. This means that essentially, you end up with different tiers of clients in open bounties, with big name clients (such as Google, Apple, Facebook) receiving a lot more attention than lesser names. The exception being, when you pay more for a private program. The impact of this is that you compete with other organisations using a bounty site, often in what’s essentially a bidding war to get more tester attention. When you compare this with penetration testing at a consultancy, you don’t have the concept of trying to outbid other clients to get more time or better testing for your project.
Bounty hunters are often specialist ‘farmers’ in a small number of unique and nuance attacks. From chatting with various bug hunters, one of the key trends I’ve noticed from the top guys, is that they normally have a handful of unique attack or recon vectors. These vectors are often wholly unique or a spin on known techniques. This means that top hunters will often try their own techniques across sites sequentially and move on quickly if they don’t have any luck. They’re unlikely to perform the typical methodology steps that are characteristic of a penetration test, again meaning that coverage has a low level of assurance. This means that as a client, you can often find that more common manually exploitable issues (that are non-trivial) can be overlooked as the top guys will move on quickly if they don’t get a pay-off within a reasonable timeframe.
Criminals can hide in the pack and sell your vulnerabilities to someone else, rather than you. Quite often, the price that can be fetched on the Black-market for a vulnerability in a popular website can be an order of magnitude higher than is offered by a bounty program. This can lead to the temptation for bug hunters to turn to the dark side. This is one of the downsides to opening up your site to a bug bounty program, as it gives a plausible deniability to a BlackHat bug hunter and invites them in the front door with access and a ‘good reason’ to be poking around. Moreover, sites such as Zerodium legally buy and sell vulnerabilities in applications to the highest bidder. There’s an interesting conversation on this topic to be had around Wassenaar and copyright laws such as DMCA in the U.S., but that’s out-of-scope for this post. Beers sometime?
Bounties often give clients what they want, rather than what they need. It’s not uncommon to see restrictions for certain types of bugs on a bounty. Quite often this covers some pretty serious issues, and even the top five of the OWASP top ten. This is obviously the client’s prerogative to run their program as they see fit. However, it’s a really bad security practice and one that you never see any self-respecting pen testing company adhere to. The reason I believe this is an issue, is because it shows risk acceptance of the most commonly found and exploited vulnerabilities on the Web. It’s like saying “I accept we have XSS on our site and I don’t care”. As simple and ubiquitous as XSS is, or indeed many other common issues are, they still represent a big risk to the end user and leaving these out of bounties perpetuates poor security hygiene at the most basic level.
It can get noisy. If an organisation chooses not to select the option to have the Bug Bounty company perform validation on their bugs before they’re recorded, they’re in for a wild ride. From speaking with a large number of organisations who’ve run their own programs or not chosen this option from a third-party, it’s clear that the skill level of bounty hunters is extremely inconsistent. Again, this is an issue with the ‘anyone can join’ model, where many of the hunters blindly run intrusive scanners across your environment in the hope that their cracked version of Acunetix picks up something juicy. Compared with professional services firms where testers are interviewed, background checked and provided with indicative methodologies (as well as indemnities!), it seems like a false economy to take this approach. One notable exception (pointed it to me by @n0x00) is SynAck, who do actually perform background checks and offer training to their hunters, kudos to them.
It won’t help you pass compliance for a penetration testing requirement. A typical crowd sourced bug bounty program will not satisfy compliance. I’m sure some of them will tell you differently, but the only way they’re telling the truth is if they’re doing penetration testing (or the auditor is easily fooled). Let’s take PCI 3.2 as an example. Requirement 11.3 contains a requirement to test applications (obviously bounties don’t deal with Network / Infrastructure testing) within it. It states that a methodology must be applied that follows certain criteria, most importantly for this example, the methodology “is based on industry-accepted penetration testing approaches (for example, NIST SP800-115)”. This is the first place Bug Bounties fall down, as they cannot provide a methodology, as each tester uses their own and I doubt any of them have created theirs based on the 80 pages of joy known as NIST SP800-115, the 213 pages of OSTMM 3 or any other legitimate standard. Mostly because, life’s too short. Secondly, the standard also states that all elements of requirement 6.5 must be covered during application testing, which requires some sort of statement attesting that this has been the case i.e. a de facto methodology.

What Bounties Are Good For

Managing a bug bounty program for your business. I would say this should be the primary focus of Bug Bounty companies. They do a fantastic job of finding nuance bugs from commonly studied applications. As previously mentioned, there are a huge amount of talented people who participate in this community already.
Recon and automation techniques. Bug Bounties have really helped push the boundaries of automation and discovery techniques within the field of ethical hacking. Shubs has been one of the major proponents of this along with Jason Haddix from Bug Crowd, so huge kudos is due to them (and various others) for this. HackerOne also does a large amount of knowledge sharing publicly, which has contributed to the education of people at all levels of the industry.
Keeping grey hats from going to the dark side (sometimes). Bounty programs have made security testing accessible globally and made it a career option for some people (especially in LEDCs), keeping them on the right side of the law.
Providing experience and opportunities for newcomers. One thing that’s been apparent to me from interviewing candidates and also socialising with the community, is that bug hunting is a great route into the industry. Shubham told me about his back story in the process of writing this post, how he went from flippin’ burgers to having two InfoSec jobs (at a consultancy and his bounty exploits) and seeking (making!) his fortune. Both Shubs and Yappare have explained to me that they owe a lot to bug bounty programs, as do a lot of others who’ve been given a route into the InfoSec world.
Marketing.

In summary, it’s my view that Bug Bounty businesses are trying to diversify into an area that do not fit their operational model. I believe that many of them are misleading clients as to the efficacy of their services and purporting to replace penetration testing and other security assessment services. I do believe there is a place for Bug Bounty programs in the InfoSec world, providing additional assurance to mature application owners who already do SDLC and Penetration testing as part of their security validation program (emphasis on validation!). However, they do not function well as a replacement for traditional penetration testing, as it’s not possible to assure (or in good conscience proclaim) proper coverage of the audit elements required for a good application security test. This cannot be built into a crowd sourced approach without first validating testing skill, methodology and most importantly motivation. In short, Bug Bounties are a ‘nice-to-have’ for those who have the money to include this approach as a security overlay.

I hope this has been a useful post, especially for those who’re considering procuring application security assessment services. I’m happy to respond to all (sensible) comments or questions, please add them below if you wish.

One of the plagues (in my humble opinion) within the InfoSec Community is compartmentalised thinking. We put different areas of our organisations into boxes and turn paradigms into silos. The artificial constructs, certification syllabi and subscriptions to schools of thought we use, structure our thinking and limiting our learning and interactions. This is not to say that any [particular] structured learning approach is wrong, but that we need to factor in the impact of these constructs on our thinking, learning and development as practitioners.

Alternative approaches to schooling have always fascinated me and made me somewhat envious of the beneficiaries. Consistently, research points towards the current approach to education as antiquated and at odds with the goals of inspiring learning and effective reasoning. It’s my belief that it’s never too late to fundamentally change your outlook and I’m consistently inspired by the innovative implementations of countries like Finland (and indeed, people I know in industry) to change the way I think, communicate and impart knowledge. This post talks about some of the different approaches to teaching / learning and how a joined-up approach can break damaging compartmentalised thinking.

Phenomenon-based Learning

The first thing I want to discuss is phenomenon-based learning. This approach was popularised by the Finnish schooling system in their National Curriculum Reforms in August 2016. The focus of ‘FBL’ is to improve the ‘authenticity’ of the learning experience, linking ideas to real-world scenarios and working in an interdisciplinary fashion.

They theory behind this approach is termed ‘constructivism’. In constructivism theory “learners are seen as active knowledge builders and information is seen as being constructed as a result of problem-solving, constructed out of ‘little pieces’ into a whole that suits the situation in which it is used at the time.”[1]. The approach also focuses on social interactions as a conduit for learning, drawing inspiration for how our understanding of language is developed. Essentially, the idea is to replicate how we learn more naturally as humans by not pigeonholing subjects and concepts.

In Finnish schools, this doesn’t totally replace taught subjects, but it augments some wider ideas and links all sorts of different facets (the common example given is the European Union) to real-world ideas. More information on the specifics of the approach can be found here.

How Are We Getting It Wrong?

Similar to the way in which school subjects neatly put ideas in pigeonholes, Information Security creates paradigms, career paths and exam tracks that shape thinking. This can create narrow fields of view and subscription to pre-conceived ideas related to a particular approach. Put simply, we’re creating worker-bees in an industry that needs to be creative and collaborative, as well as regimented. I’m not saying that any and all frameworks are damaging or have no value, but we should not allow them to narrow our sights. I see these types of structures as useful references to help guide us to make the right decisions, rather than a messianic answer to security. As an example, Agile development is something that has crept into this category and become a lifestyle brand. I think the more niche the area, the higher the chances are that it will become a cult-like framework and subsume a portion of practitioners.

While security audit is a useful tool, and frankly the only reason why a lot of organisations bother doing any security at all, it can create the idea that security and compliance are the same thing. This is destructive thinking and also creates a lot of the issues we see whilst going about our daily business.

In my view, few approaches are as effective in exposing the lack of joined-up thinking and processes than a live fire exercise (such as Red Teaming and attack simulations). Over the years, as I’ve orchestrated and run an array of attack simulation campaigns for organisations of various sizes, a few things have become clear about the players in the game (although not all, it should be said). I use the pronoun ‘we’ as I think we’re all guilty of this sometimes.

We don’t talk to each other, especially outside of our business functions.
We don’t like to concern ourselves with the roles of people outside our business unit.
We label ourselves and define our intellectual sphere based on our job function.
We measure ourselves against an arbitrary standard (often certification paths).
We think it’s someone else’s job to join the dots and think about the bigger picture.
We think it’s not possible to know everything, so it’s better to specialise and know one thing well, sometimes avoiding learning on this basis.

I believe that this type of attitude / thinking is something we learn early on and is not something easily undone. I’d be so bold to say that those who move outside this mind-set differentiate themselves, as people who’re useful in the security field vs. people who’re not. Moreover, the fundamental problem we have is the lack of quality thinkers who’re capable of creating holistic models and able to solve challenges collaboratively and creatively. I don’t think this issue is unique to security, but it’s certainly prolific and we’ve become a field that has more than its fair share of ill-suited people making up the numbers due to the skills gap. This probably sounds exceptionally harsh, but please let me explain. You could argue that this is real-life and there’s always a bell curve within a group of people. However, is this a trend that’s mirrored in other high-end vocational jobs, such as Doctors, Neuroscientists or Chemical Engineers? Is our capability as an industry proportional to the stakes and do we offer good ROI? It’s something to ponder.

The Future is Purple (and other Polychromes)

In scientific research, an interdisciplinary approach has been commonplace since the early 90’s. For example, Physics and Computing are combining to create quantum computers and genetics and neuroscience are unravelling the functionality of the brain. Should a joined-up approach be reserved for only grand scientific challenges? I don’t believe so. Setting this in the context of an organisation that is a high value target, under attack from a capable adversary, we can discuss the changes in approach and how more authentic thought processes can assist.

Phenomena-based adoption can help us re-evaluate our role in the bigger picture and help the coordinators conceptualise things in a more realistic way. We need to approach our challenges holistically and without paradigm bias. We should ask ourselves questions like: how do we know we accurately understand the real threats? If I’m a SOC analyst, do I know what a threat actor is and their MO? If I’m in the DevOps team, should I care about security outside of Availability? If I configure the Internet boarder technologies, should I care about compromises in user LANs? How does my role fit with the rest of the organisation?

Additionally, we need to consider what information we are getting following an attack and are we able to process or even understand it. Often the answer is that it’s overwhelming, and we need outside help. In this case, the most effective approach is to utilise existing expertise and knowledge and combine forces with the ‘attackers’ to understand how they do what they do and how to detect and prevent it. In the security assessment world, this is often referred to as Purple teaming (from the Red Team vs. Blue Team colour mix). This approach gives great learning opportunities for the defenders, in the problem-based style of sociocultural learning. There is also a reciprocal element as well, as the attacking team also gain a greater understanding of how difficult it is to defend. Typically, this doesn’t happen and it highlights a large issue currently in the professional services world, which is the lack of value given by professional services firms post-engagement. Collaborative phenomena-based learning is the future and purple teaming (or whatever you wish to call the idea) is key in our industry to better prepare to be more secure.

So, What Can Finnish School Children Teach Us as A Community?

I’m a big fan of bullet points and exclamation marks (and interrobangs)…

Break out of silos and compartmentalised concepts!
Don’t let curricula limit your learning!
Think critically about your role and remit within your organisation.
Seek learning opportunities outside of your role.
Security is an overlay of IT functions and human processes, therefore, it’s an advanced discipline and should be treated as such.
Security is everybody’s responsibility!
Learn in a style that is more authentic, read around topics.

Bounties Bug Me (A Little)

What Finnish School Children Can Teach The InfoSec Community