- Report this article
Hurricane Labs
Hurricane Labs
Managed Splunk and Security Services for your business's unique use case.
Published Sep 28, 2021
+ Follow
In his recent Splunk tutorial, Josh discusses different methods for anomaly detection, including standard deviation, MLTK, and StreamStats. This post provides a basic overview of his talk; to learn more about this topic, you can find the unabridged post here.
What is standard deviation?
Standard deviation measures the amount of spread in a dataset using the value’s distance from the mean. With standard deviation, a certain percentage of data will be seen as anomalous depending on the distribution of data. In security contexts, user behavior is most often an exponential distribution; in other words, having more data means more outliers–and that means more alerts.
What about MLTK?
Splunk’s Machine Learning Toolkit (MLTK) adds machine learning capabilities to Splunk, including an algorithm for anomaly detection called DensityFunction. DensityFunction, however, has limitations with large datasets.
Recommended next reads
Using StreamStats to get neighboring values
Streamstats can mimic alert investigation by calculating distance from the nearest neighbors. If the count over the past 30 days is significantly higher than previous counts, consider it anomalous. For a correlation search, we need to make sure we’re pulling in the data we want and that it’s normalized. It can also be useful to add additional metrics to filter on.
Conclusion
Base your detection method on what an outlier is in your data. If standard deviation provides those results, stick with it–but in my experience, standard deviation provides more noise than actionable results for our use cases.
Calculating distance from the nearest neighbors works well, regularly providing anomalous results. Applying this method allows analysts to focus on abnormal behavior, reducing their workload.
Looking for more details? See the extended content here!
Like
Celebrate
Support
Love
Insightful
Funny
4
To view or add a comment, sign in
More articles by this author
No more previous content
- 7 Steps to a Proactive Vulnerability Management Plan Sep 15, 2022
- First Look: Splunk 9.0 Configuration Change Logging Jun 15, 2022
- Splunk Indexer Clustering: Your Hero in the Fight Against Data Loss Jun 8, 2022
- How to Reduce Your Organization’s Vulnerability to Social Engineering May 11, 2022
- Getting Started with Automation Before You’re Ready to SOAR Apr 27, 2022
- The Russia-Ukraine War: Malware Risks and Mitigations Apr 5, 2022
- 6 Tips for Wireless Security Jan 20, 2022
- Console Wars Part 1: Hacks for Hackers Dec 20, 2021
- Ingesting a CSV file into Splunk Dec 9, 2021
- Malware Analysis Part 3: The phases and roles of incident response Nov 23, 2021
No more next content
Sign in
Stay updated on your professional world
Sign in
By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.
New to LinkedIn? Join now
Insights from the community
- Telecommunications Systems How can you ensure the 5G system test data is accurate?
- Programming Languages How do you debug and troubleshoot monitors and condition variables in complex systems?
- Operations Research What do you do with unreliable data in network analysis?
- System Architecture How do you learn from the results and feedback of fault injection tests?
- Arena Simulation Software How do you incorporate random arrivals and service times in an Arena model?
- Data Engineering What do you do if your data is at risk when using new technology?
Others also viewed
- AIOPS and Splunk Ross Parfect 3y
- PowerProtect Data Manager: Automating Virtual Machine Whitespace Reporting Cliff Rodriguez 3w
- G2X GovCon Market Research: Weekly Roundup G2Xchange 3mo
- Splunk Discovery Day Moscow 2018 Alexander Leonov 5y
- A curious case of field formatting in Splunk and Datadog Alex Gerulaitis 3y
- The Myths of Costs, Tool Consolidation and OpenTelemetry (Webinar Recap) Rob Oram 2mo
- BIG DATA ! Dirk Reinders 7y
- Fire the Detective: Transparency in Data Ken Weston 6y
- Rocana Vs. Splunk: IT Operations Management Battle Of Words Jason Bloomberg 8y
- .conf 2017 Wrap-up Ryan O'Connor 6y
Explore topics
- Sales
- Marketing
- Business Administration
- HR Management
- Content Management
- Engineering
- Soft Skills
- See All