IBM Cloud - Structured Ideas

Welcome to the idea portal for structured ideas (i.e. product feature requests) - A more integrated and automated feedback system to connect your product improvement ideas with IBM product and engineering teams.  Happy submitting!

ELK Adapter enhancement to send logs to different ELK endpoints based on Org/Space

We would like for IBM to evaluate enhancing the ELK Adapter implementation to log to multiple ELK endpoints.  We need the flexibility to send logs to the respective ELK instance (non-production applications log to a non-production ELK instance).  We would like to configure the logging destination at the space level.

  • Guest
  • Feb 16 2017
  • Returned for specifics
  • Attach files
  • Admin
    Ananda Debnath commented
    April 4, 2017 04:44

    Hello. Thanks for adding this idea to the portal.

    Would you please add some details about why you need the log destinations emanating from a single private Bluemix environment to go to multiple destinations and configured at the space level? How do you achieve this separation today? What difficulty do you run into without this ability?

  • David Nguyen commented
    April 4, 2017 13:31

    Hi Ananda, we have different ELK environments (Development, Trial, Production) that corresponds to different spaces in our Bluemix environment.  Therefore, we would like to send logs from each of the Bluemix space to the corresponding ELK environments.  At this time, Bluemix logs from all spaces are being sent to only one ELK destination/environment.  We need help to configure logs from each space to sent to the correct corresponding ELK environment.

    Thanks,

    David Nguyen 

  • Runu Rathi commented
    April 4, 2017 20:01

    Ananda, We do not have the separation at this time. We have been draining all Bluemix logs (all spaces, dev, non-prod and prod) to our non-prod BNSF ELK stack. However, this is a huge risk as the Production applications in Bluemix do not have the same guarantees of availability and message loss from BNSF ELK stack as other non-Bluemix applications who are using the Production ELK stack. The Production BNSF ELK stack is highly available and the non-prod BNSF ELK stack is not guaranteed to be highly available. Also, we have limitations where security zones (separate security zone for Prod vs non-Prod) do not allow Production application components to connect to non-prod components. Thanks, Runu

  • Runu Rathi commented
    April 4, 2017 20:05

    Another consideration is that non-prod environment is used as a load testing environment by teams resulting in abnormally large volumes of logs. So, we cannot drain the non-prod logs to Prod BNSF ELK stack as that can be risk slowing down or worst case bringing down the Production ELK stack.

  • David Spotts commented
    May 1, 2017 17:12

    Hello,

    We currently support enterprise application logging with ELK and have both production and non-production ELK environments to support that. The two environments are mirror images of each other and located in different data centers so that the non-prod environment can provide disaster recovery for the prod environment. Given ELK is our Bluemix logging solution we need the ability to log to non-prod ELK from our non-prod spaces and to production ELK from our production spaces. This is because logging in non-prod typically occurs at higher levels than prod, and, more importantly, production log data will have different retention requirements than non-prod log data.

    With the ELK adaptor’s inability to log to multiple end points we are forced to drop another infrastructure layer in between Bluemix and ELK to provide the filtering/routing necessary to ensure that log data goes to the right ELK environment. There is obviously more cost involved with the additional infrastructure, not to mention the fact that it introduces another point of failure.

    Ideally the ELK adaptor would be somewhat configurable and, at a minimum, able to route based on space name.

    Please let me know if you have any additional questions.

    Thank you!

    David E. Spotts | Manager, Technology Services | BNSF Railway | 817-352-3838

NOTICE TO EU RESIDENTS: per EU Data Protection Policy, if you wish to remove your personal information from the IBM ideas portal, please login to the ideas portal using your previously registered information then change your email to "anonymous@euprivacy.out" and first name to "anonymous" and last name to "anonymous". This will ensure that IBM will not send any emails to you about all idea submissions