In November 2022, Microsoft announced they integrated the Zeek open-source network traffic analyzer in Microsoft Defender for Endpoint. This analyzer helps Defender for Endpoint in analyzing suspicious network traffic and detect network related attacks. In April 2023 they extended the support with Zeek by including a couple of network signatures in Advanced Hunting. By doing this Microsoft allows security professionals to create threat-hunting and custom detection rules on top of the data Zeek is gathering.

In this blogpost, I wanted to write down what I learned during my research on the Zeek implementation of MDE, and how the Zeek logs are reflected in Advanced Hunting. I also tried comparing the Zeek HTTP logs with Microsoft Entra Global Secure Access (later GSA), to understand what to expect from both solutions. At the end of the post, I added some network-related queries I found interesting. Other queries might be added later on.

As described in this post, network security data can be generally classified in four main data types:

• Full content; Also called recorded traffic like PCAP files
• Extracted content; Which extracts files from the network to feed them in analysis engines
• Transaction data; Summarizes data traffic
• Alert data; Which 'judges' traffic and throws alerts like an NIDS system

When looking at the Zeek documentation, we can read that Zeek is mainly known for their Transaction data and Extracted content. By listening on a specific network interface, Zeek is able to log the transactions being done on the wire, effectively logging protocols and network activity seen in a judgment-free policy-neutral manner.

I found this very interesting information, since this gives us more insights on what MDE does with network traffic captured by Zeek. Although Zeek is also capable of throwing alerts like an NIDS does, they state in their documentation that this kind of work is best suited for packages like Snort and Suricata. This currently still leaves me wondering if this means that Defender for Endpoint uses the Zeek notice mechanism to generate their network-based alerts, or if they are using another or their own engine for this 🤔.

The connection logging done by Zeek includes a UID field, which is a unique identifier assigned by Zeek to track related activity in other Layer 7 logs on a per-connection basis.

This is a great identifier, since it can be used to pivot from a connection-based log to other layer 7 logs such as the SSL logs for example. However, when we look in advanced hunting we can see that not all logs have these UID's present:

DeviceNetworkEvents
| extend AdditionalFields = todynamic(AdditionalFields)
| extend ConnectionUid = tostring(AdditionalFields.uid)
| summarize count() by ConnectionUid
| sort by count_ desc

Additionally, certain connection ID strings are used over multiple Local and Remote IP address connections, which do not seem to be related at all in practice. This leaves me wondering if this UID field is really trust worthy in MDE data.

DeviceNetworkEvents
| extend AdditionalFields = todynamic(AdditionalFields)
| extend ConnectionUid = tostring(AdditionalFields.uid)
| where ConnectionUid != ""
| distinct ConnectionUid, LocalIP, RemoteIP
| summarize count() by ConnectionUid
| sort by count_ desc

Finally, when further drilling down on this, I found that only certain ActionTypes have an UID present. When looking at the ActionTypes you can see that this is mainly the case for L7 Zeek logs ending with 'ConnectionInspected'

DeviceNetworkEvents
| extend AdditionalFields = todynamic(AdditionalFields)
| extend ConnectionUid = tostring(AdditionalFields.uid)
| extend ConnUidPresent = iff(ConnectionUid == "", "False", "True")
| summarize count() by ActionType, ConnUidPresent

I found this unfortunate since it does not allow us to link connection based events to their related Layer7 events via Advanced Hunting ☹️. This might mean that the connection logs in MDE are not being captured via the Zeek connection log implementation 🤔.

Above things considered, I think one of the ways to really use the UID field in MDE is in addition with the LocalIP. When you use these two together, we do find logs which seem to be related to each other.

DeviceNetworkEvents
| extend AdditionalFields = todynamic(AdditionalFields)
| extend ConnectionUid = tostring(AdditionalFields.uid)
| where ConnectionUid == "<uid>" and LocalIP == "<source_ip>"

This can come in handy when you are trying to find related events during an investigation or hunting scenario.

As explained in the Zeek HTTP logging documentation, HTTP is less frequently used because of the transition to encrypted HTTPS traffic. Therefore HTTP logs is only a small fraction of the web traffic that is being logged.

Besides the fact that HTTP logs are only a small fraction, Zeek has support to decrypt TLS connections using version TLS 1.2 with the 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' cipher. This version and cipher do seem to be used the most in the environments I researched, but this is of course only a small portion of the total traffic:

// SSL inspections
DeviceNetworkEvents
| where ActionType == "SslConnectionInspected"
| extend AdditionalFields = todynamic(AdditionalFields)
| extend Direction = tostring(AdditionalFields.direction)
    , Version = tostring(AdditionalFields.version)
    , FQDN = tostring(AdditionalFields.server_name)
    , Cipher = tostring(AdditionalFields.cipher)
// Get traffic that Zeek is theoretically able to decrypt https://docs.zeek.org/en/master/frameworks/tls-decryption.html
| summarize count() by Version, Cipher
| sort by count_ desc

Sadly enough Microsoft does not seem to have implemented this feature into the MDE Zeek Implementation, since I was not able to find L7 inspected logs over HTTPS.

To have more visibility into HTTPS logs, we will need a 'Next-Gen' firewall or Proxy solution who is able to perform SSL/TLS decryption.

So to what type of data is the Zeek engine of MDE actually looking into? Since this is not fully transparent, we have to look into the Zeek documentation and the logs of MDE. By taking a look at the Zeek Logs which are supported by default, we can see Zeek has support for the following:

A lot of these log types can be found back in the ActionType of the DeviceNetworkEvents table of MDE:

• Connection
  • Failed
  • Found
  • Request
  • Success
• DnsConnectionInspected
• FtpConnectionInspected
• HttpConnectionInspected
• IcmpConnectionInspected
• InboundConnectionAccepted
• InboundInternetScanInspected
• ListeningConnectionCreated
• NetworkSignatureInspected
• NtlmAuthenticationInspected
• SmtpConnectionInspected
• SshConnectionInspected
• SslConnectionInspected

When comparing these it looks like MDE cherry-picked the most interesting data, and is not surfacing all Zeek default log types into Advanced Hunting (which is expected). Below I tried creating a table that shows which Action Types of MDE looks to be directly related to which log files default provided by Zeek.

Notice that some ActionTypes are not present in above table, such as 'IcmpSignatureInspected' and 'ConnectionSuccess'. This is because I did not find a direct match to a Zeek log file, or because the ActionType might be related to multiple logs. Also keep in mind that network traffic such as LDAP, NTLM, Kerberos, etc. are sometimes seen in the general 'NetworkSignatureInspected' ActionType.

DeviceNetworkEvents
| where ActionType == "NetworkSignatureInspected"
| extend AdditionalFields = todynamic(AdditionalFields)
| extend Signature = tostring(AdditionalFields.SignatureName)
| distinct Signature

Since Zeek is open-source and can be modified to your own needs, Microsoft might also have added some extra custom logs which are not publicly documented.

We talked about how Zeek is not able to decrypt all types of SSL/TLS traffic, and how a proxy might help with this. Even though SSL/TLS encryption is used in more protocols than only HTTP traffic, I though it would be interesting to check the logging differences between MDE 'HTTPConnectionInspected' and the GSA NetworkTrafficLogs (comparing the other MDE network logs would be comparing apples with pears). But first, what traffic do we generally expect to be logged by a Proxy solution? This blog post provides a great explanation about this, with extra context why each log matters. To summarize, we generally expect the following from a Proxy solution:

• Duration
• HTTP Status Code
• Bytes In/Out
• HTTP Method
• URL Hostname
• URL Path
• URL Query
• Content Type
• User Agent
• URL Category
• HTTP Version
• Protocol
• File Name

When looking at an MDE log with action type "HttpConnectionInspected" (which is kind of the Zeek equivalent of a Proxy log) we can see that a couple of interesting HTTP data is being logged:

For a comparison of the GSA, I prefer to refer to the theoretical table scheme of the NetworkAccessTraffic table. This is because a lot of features of GSA are at the time of writing not live yet, meaning columns in the logs are empty a lot of times. By looking at the table scheme we see that the following HTTP related data is logged:

• DestinationFqdn
• DestinationUrl
• DestinationWebCategories
• OriginHeader
• ReferrerHeader
• ReceivedBytes
• SentBytes
• ThreatType
• XForwardedFor

To get a better overview of this, I created the below table to compare MDE Zeek HTTP related data with GSA.

This means that once GSA logging is working to it's full extend, we have all expected HTTP logging data when we put together MDE and GSA (except for Content Types). Even though, it would be great to see all HTTP logging types to be integrated in GSA eventually (still some room for improvement there) 😄.

In this blogpost I wanted to add some queries which might be useful to use on the network logging MDE has. Below two resources already have very interesting queries to be used, which I definitely recommend to look into:

• https://techcommunity.microsoft.com/t5/microsoft-defender-for-endpoint/hunting-for-network-signatures-in-microsoft-defender-for/ba-p/3429520
• https://techcommunity.microsoft.com/t5/microsoft-defender-for-endpoint/enrich-your-advanced-hunting-experience-using-network-layer/ba-p/3794693

The first queries are about identifying internet-facing devices. The first query uses the DeviceNetworkEvents table and searches for incoming public scans.

DeviceNetworkEvents
| where Timestamp > ago(7d)
| where ActionType == 'InboundInternetScanInspected' 
| extend AdditionalFields = todynamic(AdditionalFields)
| extend PublicScannedPort = tostring(AdditionalFields.PublicScannedPort)
    , PublicScannedIp = tostring(AdditionalFields.PublicScannedIp)
| distinct DeviceName, RemoteIP, PublicScannedIp, PublicScannedPort, Protocol

The same can be accomplished by using the DeviceInfo table. However, with this table we can extract two reasons for the device be internet-facing. One is by looking at the public scans and the other is by checking external network connections:

DeviceInfo
| where Timestamp > ago(7d)
| extend AdditionalFields = todynamic(AdditionalFields)
| where todatetime(AdditionalFields.InternetFacingLastSeen) > ago(7d)
| extend InternetFacingLastSeen = tostring(AdditionalFields.InternetFacingLastSeen)
    , InternetFacingReason = tostring(AdditionalFields.InternetFacingReason)
    , InternetFacingLocalIp = tostring(AdditionalFields.InternetFacingLocalIp)
    , InternetFacingPublicScannedIp = tostring(AdditionalFields.InternetFacingPublicScannedIp)
    , InternetFacingLocalPort = tostring(AdditionalFields.InternetFacingLocalPort)
    , InternetFacingPublicScannedPort = tostring(AdditionalFields.InternetFacingPublicScannedPort)
    , InternetFacingTransportProtocol = tostring(AdditionalFields.InternetFacingTransportProtocol)
| summarize arg_max(InternetFacingLastSeen, *) by DeviceName, InternetFacingLocalIp, InternetFacingLocalPort, InternetFacingPublicScannedIp, InternetFacingPublicScannedPort, InternetFacingTransportProtocol, InternetFacingReason
| project InternetFacingLastSeen, DeviceName, InternetFacingLocalIp, InternetFacingLocalPort, InternetFacingPublicScannedIp, InternetFacingPublicScannedPort, InternetFacingTransportProtocol, InternetFacingReason

With this query you might get more results compared to the first query😄.

If you want to get an overview of how many ports are being exposed to the internet over time for your onboarded Windows devices, you can use below query:

// Create a base function
let base = (){ 
    DeviceInfo
    | where Timestamp > ago(30d)
    | extend AdditionalFields = todynamic(AdditionalFields)
    | extend InternetFacingLastSeen = todatetime(AdditionalFields.InternetFacingLastSeen)
        , InternetFacingReason = tostring(AdditionalFields.InternetFacingReason)
        , InternetFacingLocalIp = tostring(AdditionalFields.InternetFacingLocalIp)
        , InternetFacingPublicScannedIp = tostring(AdditionalFields.InternetFacingPublicScannedIp)
        , InternetFacingLocalPort = tostring(AdditionalFields.InternetFacingLocalPort)
        , InternetFacingPublicScannedPort = tostring(AdditionalFields.InternetFacingPublicScannedPort)
        , InternetFacingTransportProtocol = tostring(AdditionalFields.InternetFacingTransportProtocol)
};
base()
// Get the latest resport
| summarize arg_max(InternetFacingLastSeen, *) by DeviceName, InternetFacingLocalIp, InternetFacingLocalPort, InternetFacingTransportProtocol
// Join with the earliest report
| join kind=inner ( base()
    | summarize arg_min(InternetFacingLastSeen, *) by DeviceName, InternetFacingLocalIp, InternetFacingLocalPort, InternetFacingTransportProtocol
) on DeviceName, InternetFacingLocalIp, InternetFacingLocalPort, InternetFacingTransportProtocol
// Make a data point for each day between earliest and latest report
| extend Range = range(bin(InternetFacingLastSeen1, 1d), bin(InternetFacingLastSeen, 1d), 1d)
// Now expand all datapoints for dates the ports have been active
| mv-expand Range
| where Range != ""
| summarize count() by InternetFacingLocalPort, bin(todatetime(Range), 1d)
| render linechart

But if you do want to find public exposed Linux devices, the below very short query can tell you which devices have inbound connections coming from public IP addresses.

DeviceNetworkEvents
| where ActionType contains "InboundConnection"
| where RemoteIPType == "Public"
| distinct DeviceName

Do keep in mind that if your server is protected by a reverse-proxy, the RemoteIP will most likely be the the internal IP of the proxy.

Find CVE exploits on a network where the server is vulnerable to, combining the power of Network Inspection and MDE TVM data 💪.

// Get all the TVM data
let tvm_data = DeviceTvmSoftwareVulnerabilities
| distinct DeviceName, SoftwareName, SoftwareVendor, SoftwareVersion, CveId, VulnerabilitySeverityLevel;
// Get CVE signatures on the network
DeviceNetworkEvents
| where ActionType contains "NetworkSignatureInspected"
| extend AdditionalFields = todynamic(AdditionalFields)
| extend SignatureName = tostring(AdditionalFields.SignatureName),
    SignatureMatchedContent = tostring(AdditionalFields.SignatureMatchedContent),
    SamplePacketContent = tostring(AdditionalFields.SamplePacketContent)
| where SignatureName contains "CVE"
// Join the TVM data of the related device
| join kind=inner tvm_data on DeviceName
// Check if the server is vulnerable to the detected CVE in network traffic
| where SignatureName == CveId
| project-away DeviceName1