Security issue discovered: Are you performing SSL decryption with Websense? Read this.

This post has 39 Replies | 18 Followers

Not Ranked
Posts 4

Here's a problem I've uncovered.

Go to https://www.igoogle.com . This is a CNAME for www.google.com, which causes a common name missmatch incident. This is correct.

Then a second incident is created for the legit https://www.google.com certificate, which then denies all traffic. According to support/development, this is by design. Their suggestion is to delete the incdent or tunnel/allow it.

My problem is my users have HTTPS anywhere plugin, so I get A LOT of these types of redirect/denies. I also have 7 WCG hosts (one at each remote location), so manually maintaining this is not a workable solution. I'm going to raise hell for a feature request to toggle this on my content gateways.

|
Top 50 Contributor
Posts 39

Does anyone have any issues or thoughts with the way the content gateway manages its certificate store? The certificate store is visible in the WCG under Configure > SSL > Certificates > Certificate Authorities.

One issue we observed is this: There was an expired Verisign certificate, an intermediate CA (subject OU string includes the word LIABILITY) under the root CA (subject OU is Class 3 Public Primary Certification Authority). I deleted the expired cert and added the newer one (using the Add Root CA tab). However, the new cert does not show up in the certificate store. It's definitely there, because if I try to add it again, I get a message saying the issuer is already found in the database.

When I visit the web site using the cert (in this case https://wm01.harvardpilgrim.org), I no longer get an expired cert warning, which is good and correct, but the intermediate cert still does not show up in the cert store.

Tech support tells me the cert store is updated as users visit web sites. This appears to be true as our different WCGs show different certs in the cert store. I haven't done enough testing to know if the WCG is automatically adding "allowed" root CAs, but it does seem to be adding allowed intermediate CAs.

The tree structure displayed for the cert store does not make sense. There are root CAs under root CAs.

It might be nice if the certificate store separated the root certs and the intermediate certs, like Windows certmgr does. It seems Websense should consider some type of trusted root certificate program, like Microsoft has, and possibly provide a way to tie the WCG store into AD so it can be managed from AD.

When Diginotar was compormised, I had to go to all the WCGs and manually deny the certs.

 

 

|
Top 150 Contributor
Posts 23
Moderator

Thank you for your input on the Websense Web Security Gateway.  We appreciate the feedback as it helps us make the product better for all customers, and we apologize for any inconvenience you may have experienced. 

As the thread indicates, issues with SSL traffic can be a result of number of reasons – some being the inherent nature of how SSL works, some are caused by websites not adhering to standards, some may be the result of SSL-related bugs with the Web Security Gateway solution and others might represent opportunities for Websense to add new functionality to better support our users.

The Websense Engineering team is working to verify the issues you have brought to our attention, and will advise on solutions as soon as possible.  Websense is also planning on publishing a Best Practices Guide for managing SSL traffic to help customers manage access to SSL web sites more effectively. Websense will update this forum as new information becomes available about how to better manage SSL traffic.

|
Top 10 Contributor
Posts 986
Trusted Users (MVP)

Actually the recent CA compromises SUPPORTS my suggestion.  My whole point was because CA's are getting compromised left and right (another CA admitted to being compromised today in fact) the whole idea of relying on Certificate Validation as a security control is becoming less and less reliable.  How many CA's on the trusted CA list of every browser are compromised today and they haven't admitted to it yet (or know)?

If in the past year we've had about 6 CA's admit being compromised and removed from the Trusted CA list you can bet there are many more CA's on the list that are also compromised and either have kept it quiet or don't even know themselves. 

In the end I would prefer to have Certificate Validation on if I could, but due to the issues with using it I had to choose between SSL Decryption without Validation, or no SSL Decryption.  The security controls I gain through SSL Decryption far outweigh the loss of Certificate Validation so that's what I went with and would recommend others to do the same.  Any high privacy risk sites (Financials for example) I exclude from SSL Decryption anyway and that allows the browser to do validation... even if the end user won't bother reading the warning if it comes up before they hit continue anyway.

|
Top 50 Contributor
Posts 69

So, because a few CA's get compromised, that means the entire system is flawed, and therefore, useless? 

As you mention by your own admission, you would prefer to have the Cert Validation on.  I think any sensible/responsible security-aware individual would.

However, I cannot agree with your assessment that SSL decryption without validation is "okay".   I'm not happy with complacency.  It is true that I am also not using validation at the current moment, but I have not forgotten about this major glaring flaw, and I mention it to anyone/everyone every opportunity that arises.  This goes against all IT security sensibility, and if you brought this up in a conversation with your CTO stating this to be a non-issue, I'm fairly certain that may end up being a terminal career move. 

Think of it this way: You are effectively fooling everyone in believing that their SSL connection is trusted/secured, in actuality it is not.   It's a different story altogether if you do not use an internally trusted CA cert to fool everyone, but I'm assuming you do.  It poses a corporate liability.

I understand that you are between a rock and a hard place much like the rest of us, but does that mean you settle by ignoring this aspect of security altogether?  That's a very high price to pay.

Websense, please fix this!   I've been calling up support for over a year now and I'm glad I've finally written this thread, as now it's getting some due attention.  I've hit a wall with support otherwise and it's such a sloppy process.  I've been told that support has been able to replicate my issue, but yet they want to keep chasing me down for additional intrusions into my network...I believe to persuade me to just give up.

For everyone else who has stepped up and commented in this thread, thanks for making this more visible on everyone's radar...

 

|
Top 500 Contributor
Posts 8

So the Websense moderator, Yar On, suggests that this issue can be remedied by some best SSL practices that none of us are aware of or have implemented.  When can we expect to receive this Best Practices article?  How about a KB article in the mean time?

|
Top 50 Contributor
Posts 69

Update:

Squeaky wheel gets the grease...well, usually. ** Squeak Squeak Squeak **

Got a call from Jacob not too long ago, and was offered to be placed into the EAP program, as I've brick walled into this issue.  So thank you for the call.  However,  subsequent emails and calls in looking to speak to him went unheard.  I finally got a hold of the right engineering team working the website I was having so much problems with (after months of attempts), but no websense representative to mediate/address the issue now. 

Still waiting for some response, as my end user community here is definitely not forgetting about their wells fargo website woes...  and they are not letting me forget about it, either.

Also, regarding whatisadrexel posted, waiting on Yar On's link/post for Best Practice so we can verify we didn't somehow smurf up the settings.

Older case: 785987

New case: 890822

|
Top 150 Contributor
Posts 23
Moderator

Thank you again for your detailed feedback on SSL features in Websense Web Security Gateway.

The Websense Engineering team has been investigating the SSL issues you’ve brought to our attention.

This forum post provides updates for many of them.

A maintenance release for version 7.6 is being developed now, for availability in the second quarter of 2012.  This release includes key improvements for the following areas:

·         “Unknown revocation state” improvements are being made – expect significant reduction in these incidents.

·         SNI enhancements are underway.  If the SSL handshake with a server fails with SNI enabled, then another attempt will be made without SNI. This will prevent users from receiving “Common Name mismatch” errors when SNI is provided.  Websense software will be able to connect to servers that are not SNI-compliant.

·         Updates are being made to our certificate authority (CA) store, to prevent blocked sites from occurring with “explicitly denied” errors.  The changes will also help to prevent “local user not found” and “expired certificate” errors for certificates that have been renewed recently.

Websense has also prepared a CVE Best Practices Guide for the SSL Manager’s Certificate Verification Engine.  This article helps our customers to troubleshoot certificate issues and manage user access to SSL web sites. The guide is being posted on 17 February 2012 at this location:

http://www.websense.com/content/support/library/web/v762/wcg_ssl_cve/first.aspx

Following are more details about our research to date, including plans for corrections in upcoming releases:

 

“Unknown revocation state” errors: why are there so many?

 

These errors occur for a variety of reasons.  When you enable SSL Manager and enable CVE, you can set several options that specify your site’s certificate verification policies. You choose these settings in Content Gateway Manager on the page: Configure > SSL > Validation > General.

 

Some of these settings can lead to an increase in “unknown revocation state” errors. 

·         Some of this increase is due to Websense logic, so we are implementing key changes in our next maintenance release, while continuing to research additional improvements.

·         The new CVE Best Practices Guide includes additional information about the expected impacts of each CVE setting.

 

1.       If you choose the option Block certificates with no CRL or with unknown OCSP state, this can produce many “unknown revocation state” errors.  Because certificates are often missing OCSP information, a large number of errors can result.

 

Websense Action Plan: We are investigating ways to improve the logic in this feature, or offer more granular control, in the upcoming maintenance release for Content Gateway.

 

What you can do in the meantime:  Until the improvements are available, most customer sites using Content Gateway versions up to and including 7.6.2 will want to disable the option, because so many certificates are missing the OCSP data.  You can disable it in Content Gateway Manager on the page: Configure > SSL > Validation > General.

 

2.       The CVE option Check certificate revocation by OCSP can also produce “unknown revocation state” errors, as well as “Revoked certificate” errors. The number of “unknown revocation state” errors for this setting is not typically large.

 

Websense Action Plan: No changes are planned for this setting.

 

3.       “Unknown revocation state” errors can result when the Server Name Indication (SNI) extension of SSL/TLS is not properly supported by the destination server.  For these errors, the workaround is to either bypass the site or disable SSL Manager SNI support.  

 

Websense Action Plan:  An SNI enhancement is being built.  If the SSL handshake with a server fails with SNI enabled, then another attempt will be made without SNI. This will prevent users from receiving the “Common Name mismatch” errors when SNI is provided, and Websense software will be able to connect to servers that are not SNI-compliant.

What you can do in the meantime:  See this online article about “Peer suddenly disconnected” errors.  It includes the steps for turning off the SNI feature. Note that you must also disable the CVE option Deny certificates where the common name does not match the URL. If it is not also disabled, the number of “unknown revocation state” errors will increase. You can disable it in Content Gateway Manager on the page: Configure > SSL > Validation > General.

 

Certificate Authority store needs updating

Our certificate authority (CA) store will be updated in the upcoming maintenance release. This will help to prevent blocked sites from occurring with “explicitly denied” errors, when in fact the certificate is new and is valid. The changes will also help to prevent “local user not found” and “expired certificate” errors, for certificates that have been renewed recently.

Websense Action Plan: We are also researching methods for updating the CA store more frequently.  We will provide an update on this research as soon as decisions are made.

What you can do in the meantime:  You can add certificate authorities to the certificate authority tree. To do this, follow the steps in this article.

Additional updates about our SSL research and planned improvements will be posted in this forum as soon as available. 

Be sure to check the Best Practices Guide for specific SSL deployment recommendations and additional information about certificate verification.

Thank you.

 

|
Top 10 Contributor
Posts 481

Thanks for the detailed response. Hopefully this also fixes the problem with wildcard certificates showing as "unknown revocation state" even though the CRL, OCSP and certificate chain boxes are all un-checked and "Allow wildcard certificates" is checked. (v7.6.2)

|
Top 10 Contributor
Posts 2,443
Editor
Moderator

Ray Pesek:
wildcard certificates showing as "unknown revocation state" even though the CRL, OCSP and certificate chain boxes are all un-checked and "Allow wildcard certificates" is checked.

This is a bug and will be fixed in 7.6.5 due out later this year.

JACOB SLOAN, CCNA, WCSE

 

|
Top 500 Contributor
Posts 11
Editor
Moderator

We are pleased to share news here about several SSL feature improvements. These were released in May for sites using Websense Web Security Gateway.

In addition, an upcoming release for Web Security Gateway will bring further advances to SSL.

Version 7.6.5, released in May of 2012, brought the following SSL certificate verification engine (CVE) enhancements:

·         The logic for "Block certificates with no CRL URI and no OCSP URI" was improved, to be more consistent with administrator expectations. This will reduce the number of incorrect "Unknown revocation state" failures. 

·         SSL connection logic was improved for sites that do not support SNI. If the SSL handshake with a server fails with SNI enabled, then another attempt is now made without SNI. In this way, Websense software is able to connect to servers that are not SNI-compliant. And, this prevents

users from receiving "Common Name mismatch" errors when SNI is provided. Customers who had previously disabled SNI are encouraged to enable it with the new version.

·         Prior to version 7.6.5, clients using a Google Chrome browser (version 18.0.0 or higher) or a Safari browser to request SSL Web sites would receive a warning from the browser that said: "The site's security certificate is signed using a weak signature algorithm." Clients were unable to access an SSL Web site. Content Gateway now uses a more secure SHA-1 algorithm (instead of MD5) to sign SSL certificates, resulting in increased access. 

·         The certificate authority (CA) store was updated with several new CAs. This can reduce the number of "explicitly denied" errors that may result after a new CA is added to the store. 

An upcoming release (version 7.7.0) will carry forward those changes and bring these additional improvements:

Coming SSL Manager enhancements:

·         In current versions, in some browsers, opening the SSL Manager for the first time causes a certificate warning to display. In the next release (version 7.7.0) , this behavior does not occur. SSL Manager will use the same certificate as Content Gateway Manager.

·         SSL Manager will sign all dynamically generated certificates with a SHA-1 algorithm.

·         The Web Security SSL decryption bypass feature will be expanded and will support:

·         Client bypass by IP address and IP address range

·         Destination bypass by hostname, IP address, and IP address range.

A wildcard (“*”) will be accepted in the hostname, to match subdomains.

For example, “*.example.com” to match “www.example.com”, “mail.example.com”, “videostream.example.com”, and the like.


Coming SSL certificate verification engine (CVE) enhancements:

·         The certificate authority (CA) store will be completely updated in the next release. This will reduce the number of new CAs added to the store when clients browse the Web, which will reduce the number of “explicitly denied” errors that can result after a new CA is added to the store. The changes also help to prevent “local user not found” and “expired certificate” errors for certificates that have been renewed recently.

 

Websense Action Plan

Websense specialists continue to research methods for updating the CA store more frequently.  We will provide an update on this research as soon as plans are made.

In the meantime: You can add certificate authorities to the certificate authority tree. To do this, follow the steps in this article.

Be sure to check the Best Practices Guide for specific SSL deployment recommendations and additional information about certificate verification. 

And please watch for updates in the Release Notes accompanying our future Websense Web Security Gateway patches.

Thank you.

 

|
Top 10 Contributor
Posts 481

While 7.6.5 is much improved, here is an incident that got logged today:

VERIFY DENY: depth=0, CommonName "*.Hyland.com" does not match URL www.hyland.com

Notice the case difference?

|
Top 50 Contributor
Posts 69

v7.7 seems to be okay with www.hyland.com.

check it out.

Still testing but seems promising.

 

|
Top 50 Contributor
Posts 39

I'm trying to understand the difference between 7.6.7 and 7.7 in terms of SSL manager fixes/enhancements. When 7.7 came out, the release notes listed the following

------------------------------------------------------------

Certificates signed with SHA-1

SSL Manager signs dynamically generated certificates with a SHA-1 algorithm. To create and place a certificate with a stronger SHA algorithm, see the Websense knowledge base article Creating and placing a stronger SHA certificate.

Expanded SSL Decryption bypass options

The Web Security SSL decryption bypass feature now supports:
- Client bypass by IP address and IP address range
- Destination bypass by hostname, IP address, and IP address range. A wildcard (“*”) can be specified in the hostname to match subdomains. For example, “*.example.com” to match “www.example.com”, “mail.example.com”, “videostream.example.com”, etc.

Server Name Indication (SNI) connection retry

If the SSL connection handshake fails with SNI enabled (default), then another attempt is made without SNI. This provides added support when attempting to connect to servers that are not SNI-compliant and reduces the number of related errors.

SSL certificate verification engine (CVE) enhancements

- Updated trusted certificate store. The updated trusted certificate store reduces the number of new CAs added to the store when clients browse the Web, which reduces the number of errors that can result after a new CA is added to the store.
- Fewer “Unknown revocation state” errors. The logic for “Block certificates with no CRL URI and no OCSP URI” has been improved to be more consistent with administrator expectations, which reduces the number of incorrect “Unknown revocation state” failures.

------------------------------------------------------------

The recently announced 7.6.7, which is newer than 7.7,  lists the following fixes to certificate verification:

------------------------------------------------------------

- The Certificate Authority list and revocation links have been updated and the compromised DigiNotar Root CA has been removed.

- Certificate Verification bypass now functions correctly when you are using transparent proxy.

- When Certificate Verification and Verification Bypass were enabled, and the Validation block page was modified to use alternate languages, the block page would use the common name of the certificate (instead of the correct address) as the host address. This would prevent the user from bypassing the certificate error and continuing to the correct address.

- The common name check was previously case-sensitive, occasionally causing Content Gateway to incorrectly serve Certificate Verification block pages due to common name mismatch.

- Previously, a manual update to the Certificate Revocation List would fail to connect to certain Certificate Authorities.

- The SSL module could not supply all Subject Alternative Names in certificates in a transparent proxy deployment with Content Gateway. This would cause users browsing to certain sites to receive a certificate error page.

- With Certificate Verification enabled, an underscore character in the authentication name or the domain name could cause a problem. If the user received a Verify deny error and chose "Visit site anyway," the current page would simply be relaunched. This has been fixed.

------------------------------------------------------------

From tom1231's last post, it seems at least one of the certificate verification fixes in 7.6.7 (for the case-sensitive CN check) is in 7.7. Does anyone know if all the fixes are in 7.7? I'm trying to decide whether to upgrade to 7.7 for the SSL fixes, or wait until the next maintenance release of 7.7 (which I am told will be 7.7.3 bit don't have an estimate of availability).

 

|
Top 10 Contributor
Posts 481

I can't answer that question but I can tell you that we could not run SSL Manager and perform certificate validation in 7.6.2 because of all the "unknown revocation state" and other errors. We are running it with 7.6.5 without any complaints. The only thing 7.6.5 broke that we noticed was that it made things case-sensitive and generated an error that wasn't before (and shouldn't be obviously).

I'm trying to make the same decision but for ESG reasons instead of proxy reasons. The big issue we would have with moving to 7.7 is that we would need to push upgrades of the DLP endpoint agent to every desktop and server and we don't have to do that with the 7.6.7 upgrade because we're already on DLP 7.6.3

|
Page 2 of 3 (40 items) < Previous 1 2 3 Next > | RSS