What is meant by "we're the odd minority crying in the wilderness"

The words you quoted are spoken by Granger to Montag at the end of the story. The "odd minority" refers to Granger's fellow intellectuals. They are said to be a "minority crying in the wilderness."
The phrase "crying in the wilderness" originates from the Bible. It is first referenced in Isaiah 40:3.

The voice of him that crieth in the wilderness, Prepare ye the way of the LORD, make straight in the desert a highway for our God.

The phrase later resurfaces in John 1:23.

He said, I am the voice of one crying in the wilderness, Make straight the way of the Lord, as said the prophet Esaias.

Here, John is the lone voice preparing the way for Jesus Christ's ministry on earth. So, the phrase "crying in the wilderness" has two implications. The first is that it usually references a lone or minority voice/voices. The second is that the voice always prepares the way for a person, development, or movement of import. 
In Fahrenheit 451, Granger and his fellow intellectuals are the "minority crying in the wilderness." Just like the disciple John, this group of intellectuals is ignored by the majority population. This is why Granger asserts that his group has little control over the events transpiring in the world. So, the small group of nonconformists hide themselves and spend the majority of their time memorizing whole books about every conceivable world philosophy, religion, political thought, and scientific topic.
Like the apostle John, Granger's group sequesters itself and waits to emerge from hiding at the right time, ready to share the knowledge it has gleaned (and protected) for the benefit of mankind.