日志管理(6) 写一个超级简单的日志查询页面

发表于 | 分类于 ， |   热度
| 字数统计: 3.2k | 阅读时长 ≈ 18

把所有的业务日志通过scribe收集上来后，需要给项目开发人员一个查询业务日志的页面，方便他们查日志也给我们运维节约时间，我们可以用节约的时间做更多更有意义的事。写的这个页面很简单后端用的是Django，前端展示用的是基于bootstrap的Metronic模板，因为不需要权限控制也不要登录验证故没有用到数据库。
用一个业务日志查询页面来介绍下，其它业务日志查询的逻辑都一样甚至渲染的模板都几乎一样，唯一不同的是渲染模板传递的参数有小差异。
实现的功能很简单：可以查询每天最新的日志如最新100行；根据日期查询某天日志最后100行；根据一个或多个关键字查询日志并显示最新的200行结果；每次查询的结果都可以通过页面的下载按钮下载到本地供日后分析；页面上显示的结果按时间排序，最新的显示在最前面。

1、渲染模板，每个业务日志模板都由一个基础模板和各自独有的模板构成。
基础模板bash.html

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
<!DOCTYPE html> 
<!--[if IE 8]> <html lang="en" class="ie8"> <![endif]--> 
<!--[if IE 9]> <html lang="en" class="ie9"> <![endif]--> 
<!--[if !IE]><!--> <html lang="en"> <!--<![endif]--> 
<!-- BEGIN HEAD --> 
<head> 
  <meta charset="utf-8" /> 
  <title>{ {privatetitle}}</title> 
  <meta content="width=device-width, initial-scale=1.0" name="viewport" /> 
  <meta content="" name="description" /> 
  <meta content="" name="author" /> 
  <!-- BEGIN GLOBAL MANDATORY STYLES --> 
  <link href="{ {STATIC_URL}}/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> 
  <link href="{ {STATIC_URL}}/css/bootstrap-responsive.min.css" rel="stylesheet" type="text/css"/> 
  <link href="{ {STATIC_URL}}/css/font-awesome.min.css" rel="stylesheet" type="text/css"/> 
  <link href="{ {STATIC_URL}}/css/style-metro.css" rel="stylesheet" type="text/css"/> 
  <link href="{ {STATIC_URL}}/css/style.css" rel="stylesheet" type="text/css"/> 
  <link href="{ {STATIC_URL}}/css/style-responsive.css" rel="stylesheet" type="text/css"/>
  <link href="{ {STATIC_URL}}/css/default.css" rel="stylesheet" type="text/css" id="style_color"/> 
  <link href="{ {STATIC_URL}}/css/uniform.default.css" rel="stylesheet" type="text/css"/>
  <!-- END GLOBAL MANDATORY STYLES --> <!-- BEGIN PAGE LEVEL STYLES --> 
  <link rel="stylesheet" type="text/css" href="{ {STATIC_URL}}/css/datepicker.css" /> 
  <link href="{ {STATIC_URL}}/css/glyphicons.css" rel="stylesheet" /> 
  <!-- END PAGE LEVEL STYLES --> 
  <link rel="shortcut icon" href="{ {STATIC_URL}}/image/favicon.ico" /> 
</head> 
<!-- END HEAD --> 
<!-- BEGIN BODY --> 
<body class="page-header-fixed"> 
  <!-- BEGIN HEADER --> 
  <div class="header navbar navbar-inverse navbar-fixed-top"> 
    <!-- BEGIN TOP NAVIGATION BAR --> 
    <div class="navbar-inner"> 
      <div class="container-fluid"> 
        <!-- BEGIN LOGO --> 
        <a class="brand" href="/seek/"> 
          <img src="{ {STATIC_URL}}/image/logo.png" alt="logo" /> 
        </a> 
        <!-- END LOGO --> 
        <!-- BEGIN RESPONSIVE MENU TOGGLER --> 
        <a href="javascript:;" class="btn-navbar collapsed" data-toggle="collapse" data-target=".nav-collapse"> 
          <img src="{ {STATIC_URL}}/image/menu-toggler.png" alt="" /> 
        </a> 
        <!-- END RESPONSIVE MENU TOGGLER --> 
        <!-- BEGIN TOP NAVIGATION MENU --> 
        <ul class="nav pull-right"> 
          <!-- BEGIN USER LOGIN DROPDOWN --> 
          <li class="dropdown user"> 
            <a href="#" class="dropdown-toggle" data-toggle="dropdown"> 
              <img alt="" src="{ {STATIC_URL}}/image/linux.jpg" /> 
              <span class="username">SomeBody</span> 
              <i class="icon-angle-down"></i> 
            </a>
            <ul class="dropdown-menu"> 
              <li><a href="#"><i class="icon-tasks"></i> My Home</a></li> 
              <li class="divider"></li> 
              <li><a href="#"><i class="icon-key"></i> Log Out</a></li> 
            </ul> 
          </li> 
          <!-- END USER LOGIN DROPDOWN --> 
        </ul> 
        <!-- END TOP NAVIGATION MENU --> 
      </div> 
    </div>
    <!-- END TOP NAVIGATION BAR --> 
  </div> 
  <!-- END HEADER --> 
  <!-- BEGIN CONTAINER --> 
  <div class="page-container row-fluid"> 
    <!-- BEGIN SIDEBAR --> 
    <div class="page-sidebar nav-collapse collapse"> 
      <!-- BEGIN SIDEBAR MENU --> 
      <ul class="page-sidebar-menu"> 
        <li> 
          <!-- BEGIN SIDEBAR TOGGLER BUTTON --> 
          <div class="sidebar-toggler hidden-phone"></div> 
          <!-- BEGIN SIDEBAR TOGGLER BUTTON --> 
        </li> 
        <li class="active "> 
          <a href="javascript:;"><i class="icon-table"></i><span class="title">日志查询</span><span class="selected"></span><span class="arrow open"></span> 
          </a> 
          <ul class="sub-menu"> 
            <li class={ {seek01}}> <a href="/seek/aaaaa/">A日志</a> </li> 
            <li class={ {seek02}}> <a href="/seek/bbbbb/nginx-php/">B日志</a> 
            </li> <li class={ {seek03}}> <a href="/seek/ccccc/">C日志</a> </li> 
            <li class={ {seek04}}> <a href="/seek/ddddd/services/">D日志</a> </li> 
            <li class={ {seek05}}> <a href="/seek/eeeee/nginx-php/">E日志</a> </li> 
          </ul> 
        </li> 
      </ul> 
      <!-- END SIDEBAR MENU --> 
    </div> 
    <!-- END SIDEBAR --> 
    <!-- BEGIN PAGE --> 
  <div class="page-content"> 
    <!-- BEGIN SAMPLE PORTLET CONFIGURATION MODAL FORM--> 
    <div id="portlet-config" class="modal hide"> 
      <div class="modal-header"> 
        <button data-dismiss="modal" class="close" type="button"></button> 
        <h3>portlet Settings</h3> 
      </div> 
      <div class="modal-body"> 
        <p>Here will be a configuration form</p> 
      </div> 
    </div> 
    <!-- END SAMPLE PORTLET CONFIGURATION MODAL FORM--> 
    <!-- BEGIN PAGE CONTAINER--> 
    <div class="container-fluid"> 
      <!-- BEGIN PAGE HEADER--> 
      <div class="row-fluid"> 
        <div class="span12"> 
          {% block rightareadown %}
          <br>
          <h3>请选择项目</h3>
          {% endblock %} 
        </div> 
      </div> 
      <!-- END PAGE CONTENT--> 
    </div> 
    <!-- END PAGE CONTAINER--> 
  </div> 
  <!-- END PAGE --> 
</div> 
<!-- END CONTAINER --> 
<!-- BEGIN FOOTER --> 
<div class="footer"> 
  <div class="footer-inner"> 
    Copyright (c) 2014 日志查询平台. All rights reserved 
  </div> 
  <div class="footer-tools"> 
    <span class="go-top"> 
      <i class="icon-angle-up"></i> 
    </span> 
  </div> 
</div> 
<!-- END FOOTER --> 
<!-- BEGIN JAVASCRIPTS(Load javascripts at bottom, this will reduce page load time) --> 
<!-- BEGIN CORE PLUGINS --> 
<script src="{ {STATIC_URL}}/js/jquery-1.10.1.min.js" type="text/javascript"></script> 
<script src="{ {STATIC_URL}}/js/jquery-migrate-1.2.1.min.js" type="text/javascript"></script> 
<!-- IMPORTANT! Load jquery-ui-1.10.1.custom.min.js before bootstrap.min.js to fix bootstrap tooltip conflict with jquery ui tooltip --> 
<script src="{ {STATIC_URL}}/js/jquery-ui-1.10.1.custom.min.js" type="text/javascript"></script> 
<script src="{ {STATIC_URL}}/js/bootstrap.min.js" type="text/javascript"></script> 
<!--[if lt IE 9]> <script src="{ {STATIC_URL}}/js/excanvas.min.js"></script> <script src="{ {STATIC_URL}}/js/respond.min.js"></script> <![endif]--> 
<script src="{ {STATIC_URL}}/js/jquery.slimscroll.min.js" type="text/javascript"></script> 
<script src="{ {STATIC_URL}}/js/jquery.blockui.min.js" type="text/javascript"></script> 
<script src="{ {STATIC_URL}}/js/jquery.cookie.min.js" type="text/javascript"></script> 
<script src="{ {STATIC_URL}}/js/jquery.uniform.min.js" type="text/javascript" ></script> 
<!-- END CORE PLUGINS --> <!-- BEGIN PAGE LEVEL PLUGINS --> 
<script type="text/javascript" src="{ {STATIC_URL}}/js/bootstrap-datepicker.js"></script> 
<script type="text/javascript" src="{ {STATIC_URL}}/js/date.js"></script> 
<!-- END PAGE LEVEL PLUGINS --> <!-- BEGIN PAGE LEVEL SCRIPTS --> 
<script src="{ {STATIC_URL}}/js/app.js"></script> <script src="{ {STATIC_URL}}/js/form-components.js"></script> 
<script> 
  jQuery(document).ready(function() { 
    // initiate layout and plugins 
    App.init(); 
    FormComponents.init(); 
  }); 
</script> 
<!-- END PAGE LEVEL SCRIPTS --> 
<script type="text/javascript"> 
var _gaq = _gaq || []; 
_gaq.push(['_setAccount', 'UA-37564768-1']); 
_gaq.push(['_setDomainName', 'keenthemes.com']); 
_gaq.push(['_setAllowLinker', true]); 
_gaq.push(['_trackPageview']); 
(function() { 
  var ga = document.createElement('script'); 
  ga.type = 'text/javascript'; 
  ga.async = true; 
  ga.src = ('https:' == document.location.protocol ? 'https://' : 'http://') + 'stats.g.doubleclick.net/dc.js'; 
  var s = document.getElementsByTagName('script')[0]; 
  s.parentNode.insertBefore(ga, s); 
})();</script>
</body> 
<!-- END BODY --> 
</html>

每个项目还有自己独有的模板且继承bash.html,独立模板用来重写块，独立的模板几乎一样就举一例。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
{% extends "base.html" %} 
{% block rightareadown %} 
<ul class="breadcrumb"> 
  <li> 
    <i class="icon-home"></i> 
    <a href="/seek/">日志查询</a> 
    <i class="icon-angle-right"></i> 
  </li> 
  <li>
    <a href="/seek/aaaaa/">A日志</a>
  </li> 
</ul> 
<div class="row-fluid"> 
  <div class="span12"> 
    <!-- BEGIN PORTLET--> 
    <div class="portlet box grey"> 
      <div class="portlet-title"> 
        <div class="caption"> 
          <i class="icon-reorder"></i>填入必要的查询条件 
        </div> 
      <div class="tools"> 
        <a href="javascript:;" class="collapse"></a> 
        <a href="javascript:;" class="remove"></a> 
      </div> 
    </div> 
  <div class="portlet-body form"> 
    <!-- BEGIN FORM--> 
    <form action="/seek/aaaaa/" class="form-horizontal" enctype="multipart/form-data" method="post"> 
      <div class="control-group"> 
        <label class="control-label">选择查询日期(默认当天)</label> 
        <div class="controls"> 
          <div class="input-append date date-picker" data-date-format="mm/dd/yyyy" data-date-viewmode="years"> 
            <input class="m-wrap m-ctrl-medium date-picker" readonly size="16" type="text" value="" name="date"/>
            <span class="add-on"><i class="icon-calendar"></i></span> 
          </div> 
        </div> 
      </div> 
      <div class="control-group"> 
        <label class="control-label">选择要查询的日志</label> 
        <div class="controls"> 
          <select class="span6 m-wrap" multiple="multiple" data-placeholder="Choose a Category" tabindex="1" name="logdir" > 
            {% for dir in dirs %} 
              <option>{ { dir }}</option> 
            {% endfor %} 
          </select> 
        </div> 
      </div> 
      <div class="control-group"> 
        <label class="control-label">Keys</label> 
        <div class="controls"> 
          <input type="text" class="span6 m-wrap" name="keys" placeholder="输入关键字，多个关键字用分号分隔(不输入关键字则获取当天最新的100行记录)"> 
        </div> 
      </div> 
      <div class="form-actions"> 
        <button type="submit" class="btn blue">Submit</button> 
      </div> 
      {% csrf_token %} 
    </form> 
    <!-- END FORM--> 
    </div> 
  </div> 
  <!-- END PORTLET--> 
</div> 
<div class="row-fluid"> 
  <div class="portlet box grey"> 
    <div class="portlet-title"> 
      <div class="caption"> 
        <i class="icon-reorder"></i>查询结果 
      </div> 
    <div class="tools"> 
      <a href="javascript:;" class="collapse"></a> 
    </div> 
  </div> 
  <div class="portlet-body"> 
    <div class="control-group"> 
      <a href="{ {durl}}" class="glyphicons no-js download_alt"><i></i>下载</a> 
    </div> 
  <div class="scroller" data-height="550px" data-always-visible="1"> 
    { { value }} 
    {% for mes in message %} 
      { { mes }}
      <br>
      <br> 
    {% endfor %} 
  </div> 
</div> 
</div> 
</div> 
{% endblock %}

2、视图函数，接收前端页面的POST请求并根据不同的请求条件，使用日志查询函数searchKey, noKeys, readLine查询日志，然后将结果渲染后返回给前端页面。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
#coding=utf-8 
from django.shortcuts 
import render_to_response 
from django.template import RequestContext 
from os import listdir, path 
from search import searchKey, noKeys, readLine 
import time 


def seekindex(request): 
  avg = {'privatetitle': '日志查询', 'STATIC_URL': '/static'} 
  return render_to_response('base.html', avg) 
  
def seekvenus(request): 
  message = '' 
  durl = '#' 
  value = '' 
  logPath = r"/var/scribe_data/aaaaa" 
  logList = listdir(logPath) 
  dirs = [] 
  for dir in logList: 
    dirs.append(dir.replace("1_", '')) 
  dirs.sort() 
  if request.method == 'POST': 
    try: 
      date = request.POST['date'] 
    except: 
      pass 
    try: 
      logdir = request.POST['logdir'] 
    except: 
      logdir = 'nodir' 
    try: 
      keys = request.POST['keys'] 
    except: 
      pass 
    if not date: 
      date = time.strftime('%m/%d/%Y',time.localtime(time.time())) 
    if date and logdir and keys: 
      # fname 
      temp = date.split('/') 
      dd = temp[1] 
      mm = temp[0] 
      yy = temp[2] 
      fname = '/var/scribe_data/aaaaa/'+'1_'+logdir+'/'+'1_'+logdir+'-'+yy+'-'+mm+'-'+dd+'_00000' 
      # keys 
      keys = keys.split(';') 
      # seek 
      if not path.exists(fname): 
        value = "当天的日志文件不存在" 
      else: 
        dfile = searchKey(fname, keys) 
        durl = r'/static/tmp/' + dfile.split('/')[-1] 
        message = readLine(dfile, False) 
    elif logdir == 'nodir': 
      value = '请选择日志' 
    else: 
      temp = date.split('/') 
      dd = temp[1] 
      mm = temp[0] 
      yy = temp[2] 
      fname = '/var/scribe_data/aaaaa/'+'1_'+logdir+'/'+'1_'+logdir+'-'+yy+'-'+mm+'-'+dd+'_00000' 
      if not path.exists(fname): 
        value = "当天的日志文件不存在" 
      else: 
        dfile = noKeys(fname) 
        durl = r'/static/tmp/' + dfile.split('/')[-1] 
        message = readLine(dfile, True) 
    avg = {'privatetitle': 'aaaaa日志查询', 'STATIC_URL': '/static', 'message':message, 'seek01':"active" ,'dirs':dirs, 'durl':durl, 'value':value} 
    return render_to_response('aaaaa/seek.html', avg, context_instance=RequestContext(request))

3、日志查询函数。每次先将一定条件的查询结果保存到一个新的文件，然后再从查询结果中读取最新的100行显示，这个新的文件可以通过页面的下载链接上下载到本地。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
#!/usr/bin/python 
from re import search, sub 
import logging 
import commands 

def searchKey(fname, keys, user='Unknown'): 
  logging.basicConfig(filename = '/var/www/logseek/log/seek.log', level = logging.INFO, filemode = 'a', format = '%(asctime)s - %(levelname)s: %(message)s') 
  tmpdir = r'/var/www/logseek/static/tmp/' 
  resultFile = tmpdir + fname.split(r'/')[-1]+'-'+keys[0]+'-'+keys[-1]+'.log' 
  file = open(fname)
  refile = open(resultFile, 'w') 
  while True: 
    lines = file.readlines(500000) 
    if not lines: 
      break 
    for line in lines: 
      for key in keys: 
        row = search(key, str(line)) 
        if not row: 
          row = False 
          break 
        if row: 
          refile.write(line) 
  file.close() 
  refile.close() 
  keyinfo = '' 
  for key in keys: 
    keyinfo = keyinfo + ':' + key 
  logging.info(user + ':' + fname + keyinfo) 
  return resultFile 
  
def noKeys(fname, user='Unknown'): 
  logging.basicConfig(filename = '/var/www/logseek/log/seek.log', level = logging.INFO, filemode = 'a', format = '%(asctime)s - %(levelname)s: %(message)s') 
  tmpdir = r'/var/www/logseek/static/tmp/' 
  resultFile = tmpdir + fname.split(r'/')[-1]+'-'+'new100.log' 
  comand = 'tail -n 100 '+fname+' > '+ resultFile 
  try: 
    stat, proStr = commands.getstatusoutput(comand) 
  except: 
    pass 
  logging.info(user + ':' + fname + ' tail 100 lines') 
  return resultFile 
  
def readLine(fname, flag): 
  if flag: 
    comand = 'tail -n 100 '+fname 
  else: 
    comand = 'tail -n 200 '+fname 
  try: 
    stat, proStr = commands.getstatusoutput(comand) 
  except: 
    pass 
  relist = proStr.split('\n') 
  relist.reverse() 
  return relist

4、最后看到的查询日志页面
logs-terrace
一个非常简单实用的日志查询页面，如果以后还有更多需求再慢慢堆代码。

----------------本文结束 感谢阅读----------------