小日志与大日志：通过Java进行日志记录

My use case : In an hour, I get 10000 customer requests. We create an hourly log file. Each customer log is of 5K, i.e. hourly log is 50M + some system logged data. We use grepping for post log analysis and log4j for writing the logs.

我的用例:在一个小时内,我收到10000个客户请求。我们创建一个每小时的日志文件。每个客户日志为5K,即每小时日志为50M +一些系统记录数据。我们使用grepping进行日志分析,使用log4j来编写日志。

Question : Is is better to create a log file per customer, or dump all customer logs in one large hourly file along with other system related logs.

问题:最好是为每个客户创建一个日志文件,或者将所有客户日志与其他系统相关日志一起转储到一个大的每小时文件中。

Looking for :

寻找 :

An answer, which covers (in priority order ) :

答案,涵盖(优先顺序):

Performance : Overhead while creating the log (i.e. runtime log4j performance)?
性能:创建日志时的开销(即运行时log4j性能)?
Scalability : Suppose I start getting 1 M customers in an hour?
可扩展性:假设我在一小时内开始获得1M客户?
Storage Management : Multiple small files are easy to maintain or one big file at OS level?
存储管理:多个小文件易于维护或一个大文件在操作系统级别?
Log analysis : Is grepping a large log file is better OR finding a file in OS is better?
日志分析:greping一个大的日志文件是更好还是在OS中查找文件更好?

I have seen this link, java logging : multiple small file vs one big, this scenario is little different, it questions breaking into pieces based on size, while I talk about breaking based on some unique identifier.

我已经看到了这个链接,java日志:多个小文件vs一个大,这个场景有点不同,它根据大小问题分解,而我谈论基于一些唯一标识符打破。

1 个解决方案

#1

Performance : Overhead while creating the log (i.e. runtime log4j performance)?

性能:创建日志时的开销(即运行时log4j性能)?

More files are slower to write and slower to read if you need to scan them all.

如果您需要扫描所有文件,则写入速度较慢,读取速度较慢。

If you need to read just one, it's much faster.

如果你只需要阅读一个,那就快得多。

Scalability : Suppose I start getting 1 M customers in an hour?

可扩展性:假设我在一小时内开始获得1M客户?

1 Million files is a lot but either solution can work.

1百万个文件很多,但任何一个解决方案都可以。

Storage Management : Multiple small files are easy to maintain or one big file at OS level?

存储管理:多个小文件易于维护或一个大文件在操作系统级别?

More files are harder to manage, but 1 million isn't that much these days unless you have a spinning disk in which case, it could perform very badly.

更多文件更难管理,但是现在有100万不是那么多,除非你有旋转磁盘,在这种情况下,它可能表现得非常糟糕。

Log analysis : Is grepping a large log file is better OR finding a file in OS is better?

日志分析:greping一个大的日志文件是更好还是在OS中查找文件更好?

If you only have to deal with one file it's better.

如果你只需要处理一个文件就更好了。

BTW You can have a combination such as one file for each digits e.g. all user id ending in 0, and another for those ending in 1. This allows you to partition the data and control the number of files.

BTW您可以为每个数字组合一个文件,例如所有用户ID以0结尾,另一个以1结尾。这允许您对数据进行分区并控制文件数。

1 个解决方案

#1

更多相关文章

随机推荐